Testing the path to Dimensions™

This is the third in a multi-part series of posts highlighting the journey from our very first product, Scratch-it, to where we are today with Zembula Dimensions™.

(Want to know our origin story? Check out Part 1 and Part 2 in our series.)

While developing Dimensions, we tested the Smart Banner™ at the top of emails and later added the Smart Kicker™ at the bottom. We discovered that combining both significantly boosted email performance and revenue for retailers.

These tests revealed key trends in email engagement and set the stage for further personalization enhancements. Here’s what happened.

Test 1: Smart Banner

While incubating Dimensions, we thought adding Smart Banner to the top of the daily marketing email for enterprise retailers was the breakthrough we needed. All we had to do was add a comprehensive list of personalization use cases, and we would have our solution.

Test 2: Smart Banner + Smart Kicker

When one of our customers asked, “What’s next?” we thought about a test we had done with another enterprise retailer that year who didn’t want a banner at the top of the email and instead wanted to test a banner at the footer of the email.

We agreed to run the test and were shocked to see great results.

We took these learnings to our largest customer. We let them know we had positive results from replicating the long tail of use cases we covered at the top of the email and putting it at the bottom instead.

We recommended the retailer test it: Replicate the Smart Banner at the top of the email with a larger version of the banner (we called it the Smart Kicker) at the bottom that displayed the same content.

Once they agreed, we set up our standard four-week longitudinal 50/50 audience test to track revenue and transactions. What we learned over those four weeks was amazing.

Hypothesis of this second test:

We hypothesized adding the Smart Kicker might steal some performance from the Smart Banner and give it a small lift. 

We were totally wrong.

The result: Having both the Smart Banner at the top and the Smart Kicker above the footer in the retailer’s daily email added 66% more performance!

This recommendation changed our business forever.

Test 3: Trying the Smart Banner + Smart Kicker again

With this game-changing news, we decided to approach another one of our enterprise retail customers. We shared our recommendation to the other customer and the huge lift they saw by adding a Smart Kicker to the bottom of the email.

This second customer was excited about the potential and agreed to run the test. So, we replicated the Smart Banner in a larger format at the bottom of the email. 

The result: The result was unexpected. This retailer added 33% more revenue to the overall results. The bottom banner delivered one-third of the revenue performance, and the top banner delivered two-thirds. 

Testing summary

From these tests, we realized there were different results with different retailers, but the combined performance of the Smart Banner and Smart Kicker added significantly more performance than using only one or the other. 

After 10+ tests: A secret in plain sight

After these first tests with different retailers, we knew we were on to something big. But there were still plenty of questions.

In repeating the tests, we consistently found that including both the Smart Banner and Smart Kicker maximized performance over using one of the other. 

However, two other trends emerged.

Trend 1

On average, a retailer would see two-thirds of their performance from the top of the email and one-third from the bottom. Sometimes, this flipped. We assume it depended on customer DNA and their relationship with the retailer’s email program. 

In addition:

• Some retailers send emails that customers open but don’t scroll through. They only look at the top and decide to take action or move on.
  
  
• Other retailers’ emails are read from top to bottom, and in these cases customers are more likely to take action at the bottom. 

Trend 2

The two-thirds vs. one-third learning doesn’t always apply perfectly. Some retailers have very balanced performance between the top and bottom. The more we extend the long tail, the better the performance gets (We’ve validated part of McKinsey’s personalization research with this.)

Testing the Smart Banner and Smart Kicker combo gave us some amazing insights. But we didn’t stop there. In the next part of our series, learn how product recommendations started to fit into the mix.

Subscribe to our newsletter to get the latest Zembula Insights, product updates and more:

Prefer getting your newsletters through LinkedIn? We got you! Sign up for Zembula Insights for LinkedIn here.