Holdout Group (Causal Lift) — Glossary

Holdout vs treated

A split diagram showing one group receiving recovery emails and onsite nudges while a smaller held-back group receives nothing, with the gap between them labeled causal lift.

The held-back group receives nothing, so the gap measures true lift.

Why does a holdout matter?

Without a holdout, the numbers a program reports are tangled up with people who would have converted on their own. Some shoppers come back after a cart-recovery email because the email worked; others were always going to come back. You cannot tell the two apart unless a comparable group received nothing.

A common setup holds back a small share of the audience at random, often around 5%. That group gets no email, no SMS, no onsite nudge, and no retargeting on the channel being tested. After a measurement window, usually 30 days, you compare conversion rate and revenue between the two groups. The difference is the lift the treatment genuinely produced.

A worked example

Say a store runs a three-email cart-recovery sequence and 1,000 shoppers abandon their carts in May. Some of them would return without any prompting. If roughly 10% come back on their own, that 10% is the baseline, and only the recoveries above it can be credited to the emails. The holdout is what tells you where that line sits, so the headline recovery number and the real, incremental one stop getting confused.

Holdout group vs control group

An A/B test uses a control group that gets a different version of something, the current page versus a new page. The control still receives a treatment. A holdout receives nothing at all. A control answers "which version is better?" while a holdout answers "is the whole program worth running?" Reporting an A/B control as if it were causal lift is a common and costly mix-up.

How big should a holdout be?

It is a trade-off between confidence and cost. A bigger holdout gives tighter, more trustworthy lift numbers but leaves more potential revenue unserved. Below about 50 abandoners a month, the noise drowns out the signal, so it helps to run longer windows, 90 days instead of 30, before drawing conclusions.

When can you drop the holdout?

Two situations. Once a program has shown steady, holdout-measured lift over 6–12 months, its value is well established and you can recover the held-back revenue, on the understanding that you bring the holdout back whenever you change the treatment. And for trivial changes, like a button color, the cost of measuring outweighs the cost of being wrong, so a holdout is overkill. Save holdouts for treatments where real budget or revenue is on the line.

Behavioral intervention — the most common in-session use case for holdouts
In-session engagement — where many holdout tests run
Onsite conversion agent (definition) — the agent category that supports holdout measurement
Proactive chat — where holdout measurement matters most