Holdout group
A holdout group is a share of visitors or customers who do not get the agent, the message or the campaign, chosen at random. Comparing them with everyone else shows what the action added, in orders the store can count, instead of what a report claims.
What it means
Attribution tells you which orders touched a message. A holdout tells you which orders would not have happened without it. The two numbers differ, often a lot, because many shoppers who clicked a reminder would have bought anyway. A holdout is the only way to know, and it costs the store the orders it would have made in the held-back group, which is why it should be small and temporary.
Why it matters for a store
Every vendor claims revenue. A store that runs a holdout can check the claim in its own orders. It is also the honest way to decide whether to keep paying for a tool.
An example
A kitchen-appliance store on Shopify ran a 50/50 split for a week in September 2026: half the visitors saw the agent, half did not. The agent half added to cart 2.3 times as often per visitor, significant at 95%, as the proof section of our home page shows.
How the agent uses it
How it decides describes the holdout the agent keeps: part of your visitors held back, so the lift is proven in your own data. Every claim on our case studies page states whether it was controlled.
Related terms
See also Incrementality, Attribution window, Predicted customer lifetime value.
See which of last week's visitors you missed
Thirty minutes, with your store and ad accounts open. Then 30 days free. A holdout group decides: if the agent doesn't add orders in 30 days, you don't pay.
Book 30 minutes
You'll talk to one of us.