May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

When people start using Aampe, they naturally want to understand how the system is performing. And when they ask how the system is performing, they naturally understand that Aampe can’t take credit for every app visit, add to cart, or purchase that happens in close proximity to a message being sent. There’s a baseline level of activity that you can expect on any app - the amount of stuff that will happen if you do no messaging at all, and estimates of the effectiveness of a particular messaging strategy should take that baseline into account.

One common way to validate performance is with a holdout group. There are a variety of reasons why this is not a great idea, perhaps the clearest one being that it creates apples-to-oranges comparisons when you’re adapting your messaging over time. Variations on a holdout group such as a switchback holdout or a synthetic control don’t solve the fundamental problem that, when you’re constantly changing your messaging in reaction to user responses, any holdout has a very limited shelf-life: they work well for single A/B tests that have a specific start date and stop date. They don’t work well for continuous adaptation.

That’s why we developed an adaptive synthetic control group, based on the same theoretical foundation as the Coarsened Exact Matching method developed by Gary King and his colleagues at Harvard. The method works by binning users into categories and then matching users who received a message with users that occupy the same bin but did not receive a message. We designate a monitoring window for each individual message sent, and identify a user who was at a similar level of activity on the app and who received a similar message recently, but did not receive any message that would have a monitoring window that overlaps with the message for which we’re seeking a control.

This allows us to evaluate the performance of our personalization scores (the metrics we use to determine which messaging choices an individual user will respond to). 

It also allows us to calculate precise attribution estimates, which we expose in our Composer tool:

You can find more details about the method and our validation techniques in our technical paper here.

0

Related

Shaping the future of marketing with Aampe through innovation, data.

Jul 23, 2025

Schaun Wheeler

A/B tests help us see what works on average, but real users aren’t average, their motivations and contexts vary. That’s where agentic learning shines, adapting to individuals over time. The best results come when we layer the two: tests for clarity, agents for personalization.

Jul 23, 2025

Schaun Wheeler

A/B tests help us see what works on average, but real users aren’t average, their motivations and contexts vary. That’s where agentic learning shines, adapting to individuals over time. The best results come when we layer the two: tests for clarity, agents for personalization.

Jul 23, 2025

Schaun Wheeler

A/B tests help us see what works on average, but real users aren’t average, their motivations and contexts vary. That’s where agentic learning shines, adapting to individuals over time. The best results come when we layer the two: tests for clarity, agents for personalization.

Jul 23, 2025

Schaun Wheeler

A/B tests help us see what works on average, but real users aren’t average, their motivations and contexts vary. That’s where agentic learning shines, adapting to individuals over time. The best results come when we layer the two: tests for clarity, agents for personalization.

Jul 17, 2025

Schaun Wheeler

Asking why an agent made one decision is like asking a human. The answer will sound right but miss the deeper truth. The real insight comes from system-level questions about patterns and priorities. Treat agents like teammates: give clarity, not interrogation.

Jul 17, 2025

Schaun Wheeler

Asking why an agent made one decision is like asking a human. The answer will sound right but miss the deeper truth. The real insight comes from system-level questions about patterns and priorities. Treat agents like teammates: give clarity, not interrogation.

Jul 17, 2025

Schaun Wheeler

Asking why an agent made one decision is like asking a human. The answer will sound right but miss the deeper truth. The real insight comes from system-level questions about patterns and priorities. Treat agents like teammates: give clarity, not interrogation.

Jul 17, 2025

Schaun Wheeler

Asking why an agent made one decision is like asking a human. The answer will sound right but miss the deeper truth. The real insight comes from system-level questions about patterns and priorities. Treat agents like teammates: give clarity, not interrogation.

Jul 14, 2025

Schaun Wheeler

Aampe focuses on continuous learning and alignment, not one-off wins. Like great teams, it builds success through small, consistent improvements—adapting in real time to each user's needs.

Jul 14, 2025

Schaun Wheeler

Aampe focuses on continuous learning and alignment, not one-off wins. Like great teams, it builds success through small, consistent improvements—adapting in real time to each user's needs.

Jul 14, 2025

Schaun Wheeler

Aampe focuses on continuous learning and alignment, not one-off wins. Like great teams, it builds success through small, consistent improvements—adapting in real time to each user's needs.

Jul 14, 2025

Schaun Wheeler

Aampe focuses on continuous learning and alignment, not one-off wins. Like great teams, it builds success through small, consistent improvements—adapting in real time to each user's needs.

Jul 3, 2025

Schaun Wheeler

Most attribution asks if a campaign worked. Aampe asks who it aligned with, driving outcomes by matching content to individual user preferences, moment by moment.

Jul 3, 2025

Schaun Wheeler

Most attribution asks if a campaign worked. Aampe asks who it aligned with, driving outcomes by matching content to individual user preferences, moment by moment.

Jul 3, 2025

Schaun Wheeler

Most attribution asks if a campaign worked. Aampe asks who it aligned with, driving outcomes by matching content to individual user preferences, moment by moment.

Jul 3, 2025

Schaun Wheeler

Most attribution asks if a campaign worked. Aampe asks who it aligned with, driving outcomes by matching content to individual user preferences, moment by moment.

Load More

Load More

Load More

Load More