Jun 4, 2024
Schaun Wheeler

Introducing Counterfactual Models: A Deeper Look at Message Timing

Jun 4, 2024
Schaun Wheeler

Introducing Counterfactual Models: A Deeper Look at Message Timing

Jun 4, 2024
Schaun Wheeler

Introducing Counterfactual Models: A Deeper Look at Message Timing

Jun 4, 2024
Schaun Wheeler

Introducing Counterfactual Models: A Deeper Look at Message Timing

Each user gets an AI Agent

Our agents fully personalize message timing - each user gets his or her own programmatic agent, and that agent decides what day of week, what time of day, and how frequently to message (plus lots of other stuff).

Why not send at a general best time?

It's not uncommon to look at aggregate results and note that one or two days and times tend to perform better than the others - Friday at 9:00pm gets only a 0.11% purchase rate, but Sunday at 6:00pm gets a 0.21% rate. "Why," our customers sometimes ask, "don't the agents just send all of the messages on Sunday at 6:00pm?"

That's a fair question. The table below is the answer. The observed purchase rate is what the rate *was*, but that doesn't tell us what the rate *could have been* if the agents had made different choices.

__wf_reserved_inherit

So we built a model to estimate those counterfactuals.

I looked at when agents actually messaged users, and then joined in a bunch of the weights that our agents used to make the timing decision, then bumped all of that data up against the weights we had at that time for all of the *other* timing options the agents could have chosen. The resulting counterfactual model estimates what would have happened if we'd sent all messages at just one time rather than personalizing.

The columns in the "actual" section of the table shows the actual number of messages and responses (purchase), along with the rate (responses / messages). The "counterfactual" section shows model estimates under the assumption that we send everything at just one time. The "comparison section" shows the difference in counts between the summed responses of the actual section and the responses for each row of the counterfactual section, and the percent difference in the two sections' rate columns.

The results

While there were two cases where our actual rates were a little lower than the counterfactual rates, in every case, the total number of expected responses was between 322k and 1.8 million purchases less than what we actually got by letting the agents personalize timing. If we average the difference, personalization does 20% better - equating to 1.1 million more purchases in this case - than just picking the "best" time.

There is no general "best" time.

Each individual user has a best time, and some users happen to overlap in their preferences. If you're picking a single or even a handful of "bests", you're missing out on tons of value.

0

Related

Shaping the future of marketing with Aampe through innovation, data.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 29, 2025

Schaun Wheeler

Discover how Aampe's semantic-associative agents adapt to varying user contexts, from fully known to unknown users, ensuring personalized experiences through continuous learning and contextual imputation.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 28, 2025

Schaun Wheeler

Explore how Aampe's semantic-associative agents differ from traditional multi-armed bandit models. Learn about their multi-dimensional action space and non-ergodic learning approach that tailors user experiences without generalization.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 26, 2025

Schaun Wheeler

Traditional customer engagement tools often constrain strategies by bundling orchestration and analysis within fixed campaign structures. Adopting an agentic approach—separating orchestration from analysis—enables more dynamic, user-centered communication, allowing for nuanced decision-making and broader impact.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

May 22, 2025

Schaun Wheeler

Agentic systems are designed to operate within a manageable state space, focusing on relevant variables to make effective decisions. This approach contrasts with traditional AI models that attempt to process vast amounts of data, often leading to inefficiencies and suboptimal performance.

Load More

Load More

Load More

Load More