Yesterday I wrote about situations where A/B tests can still be essential, even if you’re committed to an agentic-learner approach. Let’s expand on that a little. Take a look at the screenshots below. I’ve added a third notification opt-in workflow (“C”) to the two variants I discussed in my previous post:
A: Standard OS popup.
B: Pre-permission explainer screen, then OS popup.
C: Same two-step flow as B, but with personalized content in the explainer screen, dynamically generated by an agentic system.
Our A/B test is testing the idea that an explanation of the benefits of notification enablement will make people more willing to enable notifications. But that raises the question - what are the benefits? Those won’t be the same for everyone.
"Stay in the loop when new content drops."
"Be the first to know when your friends interact with your posts."
"Track your progress and get reminders to stay on target."
"Get notified when something needs your attention."
"Keep your account safe with instant security alerts."
"Hear about exclusive offers before anyone else."
"Get personalized tips to make the most of the app."
"Avoid missing out on limited-time opportunities."
"Get reminders tailored to your habits and routines."
Users are different. What motivates one person might actively annoy another. Traditional A/B testing assumes that we can find one message that works on average across everyone. But in reality, preferences shift, contexts vary, and behavioral signals are noisy Those are situations where agentic learning really shines.
So we don’t replace A/B testing with agentic learning. We layer them. The method are collaborators, not rivals. When we have limited chances and need clear aggregate signals, we should use A/B testing or bandits. When we need room to explore and adapt over time, we should bring in agentic learning.
It’s about knowing which kind of learning the situation allows.
