Aampe has partnered with Charge Running, the Live Audio Running mobile app, to personalize how running fits into the daily lives and routines of runners all over the world. In just the first month of using Aampe, Charge Running expanded its push notification template library from a few original templates to over 300 personalized messages. These messages convey value at every stage of the running lifecycle, from inspiration, to run planning, to celebrating progress and achievement.
Charge Running uses Aampe to generate personal scores for each of its runners, and automatically trigger and manage personalized messaging. During the months of July and August 2021 the personalized scores generated for Charge Running increased the activation lift for notifications between 97% and 227% over baseline. And the partnership between Charge Running and Aampe is just getting started.
The Aampe team is beyond excited to help grow the value Charge runners are getting from the best Live Running app in the world. In the rest of this post, we want to tell you a little more about our partnership.
The Value Proposition(s) of Running
According to a 2020 report from the Sports & Fitness Industry Association, about 50 million Americans (15% of the population) run or jog in some form. Part of the beauty of running is the myriad reasons someone might do it. Some people train for personal achievement at a specific running event. Some use it as a general form of exercise, while others pursue a specific health goal like weight loss or aerobic endurance. Some of us run to reduce stress or increase self-confidence. Others run as part of a game or sport, or for company and companionship, or just to get away and be alone for a little while.
Those of us who run do so at different times and on different schedules. We might have a regular morning run, or climb onto the treadmill for 30 minutes around lunch, or clear our head with a run at the end of a long day. We all fit running into our own lives in the way we each decide is best, whether as part of a daily routine or something we do on the weekends or during certain seasons. We adjust our habits and schedules to the world around us.
How Product Marketers and CRM professionals can serve deeply diverse user populations
Running apps aren’t special in the diversity of their users. As product managers, product marketers and CRM professionals working on a mobile app, it can sometimes seem impossible to reconcile the sheer diversity of the people who use our app with how we message and communicate with them. Something like Segmentation (e.g. “Casual Runners”, “Advanced Runners”, and “Athlete Runners”) just doesn’t even begin to really address diversity.
Aampe is the first messaging tool for mobile apps to help you achieve the seemingly unachievable goal of truly personalizing how you communicate with your users and customers.
Continuously Personalized Timing
Most push notification tools let you set up message campaigns. Campaigns are really rules: send certain messages to certain users over a certain time period with the goal of achieving certain results.
Aampe doesn't make you set predetermined rules. You could spend months surveying user opinion and analyzing user behavior without coming anywhere close to feeling confident that you should send a notification at 6pm on a Thursday versus 8pm on a Wednesday (or 6pm on Wednesdays, or 6pm on both Wednesdays and Thursdays, etc.)
When you first attach Aampe to your app, it doesn't know any best time to send a particular notification to a particular user. So the system allocates individual users to every possible time window on every possible day of the week. Some of those users respond by opening the app to do things. Aampe monitors those responses and learns from them. If you want more details on how that learning happens, take a look at A User Story, which provides a friendly, non-technical overview of our algorithms.
User responses to notifications get algorithmically consolidated into personalization scores - a summary of what the system has learned from previous messages. An Aampe personalization score is a value between 0 and 1 that estimates the probability that a user will accomplish one or more of the goal events (such as going for a run in the app) given that we messaged them at the particular time indicated by the score.
In other words, the personalization score for Monday at 8:00am is the probability that a user will go on a run if we message them on Monday at 8:00am. Every user gets their own personalization score for every possible choice we can make about notification timing.
Bringing value to Charge Runners
For every week Aampe managed notifications, each Charge runner got a timing assignment, and for every timing assignment for every runner, we had a personalization score that told us how good a bet that assignment was.
The following charts show how those bets turned out. One group (the orange line) activated, meaning they came to the app after we sent them the notification. The other group (the gray line) did not activate - we sent them the notification but didn’t see them afterwards. The lines are cumulative percentages, so both lines rise up to 100% as they go to the right.
Look at the top-left chart, showing results from the week of August 2, 2021. The grey line marks around 10% of non-activated users just before the middle of the chart, and almost 80% right after the middle. That means most of the users who didn’t activate had a personalization score near 0.5, which indicates that our system didn’t really know if the preference it has assigned to them was good or not. On the other hand, for users who did activate, the jump comes at the right side of the graph, right around a personalization score of 0.9. So, after the first round of learning, the system had a decent amount of confidence that certain options worked well for some users, but had little confidence that it had found options that didn’t work well.
Remember, these personalization scores are those Aampe assigned to users before the algorithm decided on the message - they were the basis upon which the system made the choice about when a notification should be sent. So the difference between the grey line and the orange line is a measure of how good a job Aampe did of predicting the right timing for each user.
After a couple weeks, we see the model becomes much more confident about both groups - that jump in the grey line moves from the 0.5 mark to the 0.0 mark - around 70% of those users who didn’t activate had received an assignment for which their personalization score was 0.0. That was too stringent, and our system automatically recognized that, the next week assigning a much smaller percentage of users timings for which they scored low. Over the last two weeks, the system continued to learn and correct, putting some users back near the 0.5 mark indicating that it really didn’t know what choice to make for them. However, most users moved to the two extremes - either the system identified a definite set of preferences, or at least identified a definite set of aversions.
Estimating lift
The distributions of scores for activators and non-activators gives Aampe a useful way to measure the performance of its learning models.
Model performance is often measured in terms of accuracy: predict which users are going to activate, and then see which users actually do. You would also look at things like your false positive rate (the model said they’d activate but they didn’t) or your false negative rate (the model said they wouldn’t activate but they did). These measures often get formalized in metrics like precision and recall.
Those measures are important, and Aampe monitors them to make sure the models that are generating an app's personalization scores are doing a good job of prediction in general. However those measures don’t really answer the most important question - did the choices Aampe made about when to message someone actually get that person to act?
The distribution charts in the previous section provide a way to answer that question. Here's approximately how it works:
- Randomly pick a bunch of users - say, around 10,000. (You can randomly pick the same user more than once - it all amounts to the same thing in the end. This helps with apps that don’t have large numbers of daily active users).
- Calculate the activation rate for users you randomly picked.
- Repeat the process, this time making a user more likely to be picked if they have a higher personalization score.
- Divide the second number by the first number and subtract 1.0.
The resulting number is lift - the percentage increase in activation you got by having Aampe manage your notifications. You can see how this turned out for Charge Running here:
From the start, using Aampe to manage notifications resulted in more activations. The blue line is “unweighted” lift, which follows the process outlined above. However, that’s just a measure of raw activation, which may or may not reflect the goals and priorities the algorithm was specifically tuned to look for. So if we repeat the process for calculating lift, and this time calculate the activation rate weighted by the model weight (indicating the extent to which the users did things that matched Charge Running's goals). That gives us the orange line.
For Charge Running, this “weighted” lift showed more clearly that Aampe was getting better at figuring out users - there’s a clear upward trend. Each lift measure is useful in a different way. Unweighted lift is a better way to answer the question "how many more people returned to the app because Aampe handled the notifications?". Weighted lift is a better way to answer the question "how much more did people run using the app - the app actions that were prioritized - because Aampe handled the notifications?"
The difference between the two lift measures shows that the notifications Charge Running sends aren't just bringing people back to the app generically- they are bringing users back to engage in specific, desired behaviors, like signing up for runs and completing them in the app.
Notifications are really just Product - and Product is about bringing users value
Charge Runners don’t run for the sake of running itself. They run to lose weight, or gain speed, or beat their record, or spend time with a friend, or feel better about themselves. Ultimately, every one of those purposes (and all the others you can imagine) - each of them a specific value proposition of running - is about that last one in the list: feeling better.
The Charge Running app is designed to help its customers run better and feel better about their running so they can feel better about their lives. That’s an ambitious and inspiring purpose to have. Charge Running chose to use Aampe to ensure that its notifications bring personalized value to its users: sending the right content, to the right users, at the right time. The result is greater engagement, and happier users.
