May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

May 3, 2023
Schaun Wheeler

New on arXiv: validating Aampe performance with adaptive synthetic control groups

When people start using Aampe, they naturally want to understand how the system is performing. And when they ask how the system is performing, they naturally understand that Aampe can’t take credit for every app visit, add to cart, or purchase that happens in close proximity to a message being sent. There’s a baseline level of activity that you can expect on any app - the amount of stuff that will happen if you do no messaging at all, and estimates of the effectiveness of a particular messaging strategy should take that baseline into account.

One common way to validate performance is with a holdout group. There are a variety of reasons why this is not a great idea, perhaps the clearest one being that it creates apples-to-oranges comparisons when you’re adapting your messaging over time. Variations on a holdout group such as a switchback holdout or a synthetic control don’t solve the fundamental problem that, when you’re constantly changing your messaging in reaction to user responses, any holdout has a very limited shelf-life: they work well for single A/B tests that have a specific start date and stop date. They don’t work well for continuous adaptation.

That’s why we developed an adaptive synthetic control group, based on the same theoretical foundation as the Coarsened Exact Matching method developed by Gary King and his colleagues at Harvard. The method works by binning users into categories and then matching users who received a message with users that occupy the same bin but did not receive a message. We designate a monitoring window for each individual message sent, and identify a user who was at a similar level of activity on the app and who received a similar message recently, but did not receive any message that would have a monitoring window that overlaps with the message for which we’re seeking a control.

This allows us to evaluate the performance of our personalization scores (the metrics we use to determine which messaging choices an individual user will respond to). 

It also allows us to calculate precise attribution estimates, which we expose in our Composer tool:

You can find more details about the method and our validation techniques in our technical paper here.

0

Related

Shaping the future of marketing with Aampe through innovation, data.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

Load More

Load More

Load More

Load More