Two click rates sat a fraction of a point apart, with the existing messaging slightly ahead, and the result was read as a verdict on the agents. The gap was too small to mean anything statistically, but the more important detail was somewhere else. The existing system had kept the abandoned-cart reminders, the birthday offers and the incentives, while the agents worked with general messages. The holdout group had compared two different jobs and reported one result.
The same pattern has come up across many accounts over the last few years. Teams want proof before they commit, and a head-to-head test is the obvious way to get it. A well-built holdout group can show whether per-user agents are changing behavior, and most of the work lies in how the two sides are set up.
A holdout group has to compare like with like
Random assignment makes the users comparable, and the messages still need matching by hand. The useful comparison is message type against message type, such as abandoned cart against abandoned cart or incentive against incentive. That means tagging every message on both sides by what it is for. Anything only one side was allowed to send stays out of the comparison. The finer those comparison tags, the more the test can say, and a broad bucket like “triggered messages” says very little.
There is a subtler difference too. In a rule-based system, one more campaign reliably adds a few clicks, because it goes out to everyone in the audience. In an agentic system, a new one-off message joins an inventory that each user’s agent chooses from. It may barely circulate until the agents have a reason to use it. Counting campaigns on each side misreads both systems.
A diagram of a fair holdout group. Top row: the existing system keeps its cart, birthday and incentive messages while the agents get only general messages. Bottom row: each message type is matched to the same type on the other side, with the orange star on the matched pair.
A holdout group comparison only means something once each message type is matched to the same type on the other side.
Small gaps are usually noise
One of our team ran a simple simulation to show how much chance moves these comparisons. Users were split 80/20 between two identical groups. The larger group posted the single highest value about 80 percent of the time, only because it had more draws. Below the extremes, either group came out ahead about half the time, though neither group was better.
Charts need the same caution. On a zoomed-in axis, a small drop can look alarming. On one account it turned out to be a move of a couple of tenths of a point. Before anyone reads a winner from a holdout group, the team should know how big a gap random assignment produces on its own.
Name the outcome before you claim it
Our own team got this wrong once, on an early analysis that first looked like a measure of users leaving the app. On review it was really measuring inferred opt-outs. Users messaged only by the agents opted out at a lower rate than users who also got the existing messages. App retention looked about the same across the groups, so we claimed the opt-out result and nothing more. Fewer opt-outs still matters, because a reachable user is a user the agents can keep learning from.
The base matters as much as the label. On another account, silent uninstalls inflated the count of users who could be messaged. Any frequency pattern measured against that base would have been spurious, so the analysis showed both definitions side by side.
Our default is an outcome that happens often and responds quickly. In practice that means a visit to the app within a short window after a message, measured daily for each comparison tag. Each side’s rate is then compared against what users in a no-contact holdout group did in the same window. Rare purchases and constructed metrics like lifetime value tend to leave the test inconclusive. Too much else changes at the same time.
Why teams ask for a holdout group first
When a team asks for a holdout group before anything else, the reasons usually fall into three groups. Some need to show that the agents beat the current approach. Some want learning they can share across the company. Many are carrying over the habit of campaign measurement, where every launch came with a test.
That habit has a real cost for the agents being tested. A team focused on the experiment can spend months on the design, while the content the agents need never gets built. The test then measures agents with a small vocabulary, which defeats its own purpose. Some businesses add one more complication. In a two-sided marketplace, treated and control users can compete for the same supply, so one group’s success can shrink the other’s.
What a fair holdout group needs
The setup is unglamorous, and most of it has to be in place before the first message goes out. A hash assigns users at random, so new users land in the right group on their own. A small no-contact holdout group anchors the baseline. Group boundaries stay fixed for the whole test, and nobody moves between groups. The agents need a content library with several versions per label, balanced across labels and covering more than one or two use cases. The existing system’s send logs, with user IDs and timestamps, let the team tag its messages too. Short-term, frequent outcomes come first, and longer-term ones follow once the test has run long enough.
The comparison also has to run over time. A rule-based system performs about the same in week eight as in week one, and the agents should keep improving. Results also don’t scale in a straight line once a team merges the split. We’ve covered the reasons in why results don’t simply double after a 50/50 split. A fair holdout group shows whether the agents are learning, and for whom the content is aligning well enough to drive action.
Schaun Wheeler writes regularly about measuring agentic systems. There is also a short video on what most marketers miss about A/B testing and AI, and more conversations like this are on our YouTube channel.
