GUEST POST: What happens when you actually run an incrementality test?

- Topic
- Analytics
- Source
- InternetRetailing
- Published
This summary was written by ShopDigest AI from InternetRetailing’s report, by Mees Frima. Read the full story ↗
Mees Frima, partner success manager at Billy Grace, explains why incrementality tests are not to be feared.
Overview
The article explains how incrementality testing, such as geo-holdout tests, reveals the true contribution of advertising channels beyond platform-reported ROAS. It argues that running these tests is less risky than feared and provides actionable insights for budget allocation. A case study shows a workwear brand reallocating over 10% of its ad budget after discovering non-incremental spend.
Key Details
- Geo-holdout tests pause a channel in one region and compare conversions, revenue, and traffic against a control region.
- Google spend typically shows incrementality in conversions; Meta often shows it in traffic rather than immediate conversions.
- A workwear brand found a meaningful portion of channel-attributed conversions came from organic traffic, leading to a 10%+ budget reallocation.
- A Meta test showed stable conversions but falling traffic, indicating the channel built demand upstream despite not converting immediately.
- The main challenge is not methodology but shifting the client-agency trust from platform-reported numbers to test-based evidence.
Why It Matters
Incrementality testing helps merchants and agencies identify which channels genuinely drive revenue versus those that merely appear effective. This enables smarter budget allocation and stronger client relationships built on evidence. The article encourages starting small and expanding the methodology to build trust over time.
Frequently asked questions
What is an incrementality test in advertising?
An incrementality test measures the true causal effect of an advertising channel by comparing outcomes when the channel is active versus paused. A common method is a geo-holdout test, where the channel is paused in one region while running normally elsewhere. The difference in conversions, revenue, or traffic between the regions reveals the channel's genuine contribution.
How does a geo-holdout test work?
In a geo-holdout test, you pause advertising for a specific channel in one geographic region while continuing it in others. You then measure the difference in key metrics like conversions, revenue, and traffic between the paused and active regions over the test period. The gap between expected and actual outcomes in the paused region indicates the channel's incrementality.
What did the workwear brand case study show about incrementality?
The workwear brand had strong organic presence, and the test revealed that a meaningful portion of conversions attributed to a paid channel came from users who would have found the brand organically anyway. As a result, the brand reallocated more than 10% of its total online advertising budget to channels with stronger incremental impact. The tested channel was right-sized rather than eliminated.
Can incrementality testing show that a channel is working?
Yes, incrementality testing can validate a channel's effectiveness. In one example, a Meta test showed stable conversion numbers but a drop in site traffic during the holdout period. This indicated that Meta was driving traffic that built familiarity and converted later through other channels, so the channel was genuinely earning its budget despite not showing short-term conversion incrementality.
Why do agencies avoid running incrementality tests?
Agencies often fear that the test results might reveal that a channel is not delivering as reported, which could disrupt client relationships and budget decisions. The article notes that the fear of running the test is usually bigger than what the test actually reveals. Overcoming this fear requires building trust incrementally, starting with small tests and expanding as clients see value.

