Avoiding the New Subscriber Trap

We’ve discovered something surprising about longitudinal testing — they’re often run incorrectly.

In email marketing, accurate test data is critical to making informed, revenue-driving decisions. The gold standard for measuring a campaign’s effectiveness over time is longitudinal holdout testing, which compares one segment of subscribers exposed to a campaign against a holdout group that isn’t.

However, one common oversight often skews results: how new subscribers are treated during testing. This mistake can lead to incorrect conclusions.

Here’s how this mistake happens, why it matters, and how to structure your holdout test.

The longitudinal holdout test challenge

From running tests across multiple enterprise retailers, we’ve seen how email lists dynamically change. How new subscribers are handled in testing can make or break its accuracy.

Almost universally, we see brands run longitudinal holdout tests where the holdout audience is created at the beginning of the test, and then as the test runs, all new signups are added to the test group. Occasionally, we see the reverse.

Both of these errors will give a false result.

Why does it matter where new subscribers go?

We’ve consistently found new subscribers are often the most engaged on your list. They’ve signed up, indicating high interest and intent to buy. Assigning these high-potential new subscribers to only one group creates an unfair advantage.

Here are how these two longitudinal test scenarios play out.

Scenario 1: Small holdout groups (5-30%)

In this scenario, you set up a holdout group that comprises a small percentage of your list — typically 5-30%. Over the four-week test period, all new subscribers are funneled into the test group.

The test results look great. In fact, possibly too great. When analyzing the results, the test group appears to outperform the holdout. But in reality, you’ve added all the most likely buyers to this group over the last four weeks. 

They’ve spent more time with your brand than the other holdout group, which didn’t see the campaign.

It’s not that your campaign drove the performance; you just benefited from new subscriber spending.

Result of test 1

The test group’s revenue performance uplift may look strong on paper but often reflects new subscriber activity, not campaign impact. Taking a small holdout group may also not achieve a high enough level of statistical significance to have confidence in the outcome.

Scenario 2: Balanced holdouts (50/50)

In this case, the email list is split evenly between the test and holdout groups. However, all new subscribers are still assigned to the test group, so the group with the new subscribers has an inherent revenue advantage.  

Whether it’s the test group or the holdout group, one group’s results will be inflated, leaving you without a clear answer on your email strategy’s effectiveness.

Result of test 2

The test results remain skewed, with inflated performance in whichever group receives new subscribers, creating inaccurate findings.

The right way to run a longitudinal holdout test

To maintain balanced audiences for the longitudinal test period, you must proportionally add new subscribers between the A and B groups as they sign up.

At Zembula, we’ve designed a streamlined solution that makes this process simple, effective, and automated. Rather than manually adding new subscribers, our approach assigns them to the A or B group the moment an email opens based on a ratio you set (usually 50/50).

This way, you get balanced groups without the manual work. We’ve tested this across multiple retailers and found more reliable results. This method of executing longitudinal holdout tests is considered the gold standard because it demonstrates statistical significance, so it’s the best way to test campaign efficacy.

Benefits of assigning groups at email open:

1. Accuracy through engagement: You only build your audience with people opening your email, which reduces the noise and provides a more accurate result.

2. Automatic new subscriber distribution: We automatically handle and distribute new subscribers between the holdout groups. They’re assigned to a group at random when email is opened to meet the appropriate ratio.

3. More testing flexibility: You can run longitudinal content tests within your daily email simultaneously because those tests are being run at the content level.

Avoiding common audience holdout test pitfalls gives you a more accurate view of campaign performance. At Zembula, we’re dedicated to helping marketers gain real insights with statistical significance to improve decision-making and drive incremental revenue results.

For more insights on effective email testing strategies that drive revenue lift, subscribe to our newsletter.

Prefer getting your newsletters through LinkedIn? We got you! Sign up for Zembula Insights for LinkedIn here.