In the analysis of the initial round of Sugarshots campaign testing, Doug Schumacher takes a look at the responses to different messaging strategies.

Campaign Details: Client: Sugarshots, Inc. Agency: Basement, Inc. Ad Network: 24/7 Real Media Ad Serving + Tracking: Atlas DMT Site Analytics: Think Metrics   View entire campaign    Tracking Report:

Strategic Recap:  We've set up the tests in phases, with a total of four phases. Each phase can be reviewed in the introductory Sugarshots article.

As our first phase is to establish a strategic foundation, the two tests within that phase look at the response rates that different creative messages generate, as well as how different audiences react to the different messages.

This Week's Test

Our goal with this first test is to determine which of different messaging strategies will produce the best response rates. We kept the ads very straightforward, trying not to skew the results with random creative variables. We also employed static Gif ads, rather than using animations or Flash capabilities. This understandably reduces response rates, but for the sake of testing, it also reduces variables and produces more reliable results. We'll add creative juice later.

We had several questions going into this test. The first was based on the relatively new category of liquid sugar. The question being: Will the term liquid sugar resonate, or will we need to reference granulated sugar for contextual relevance?

Secondly, there are several related benefits: Taste and dissolvability. Dissolvability is a key motivator for some users of artificial sweeteners, and it also provides a clear, tangible benefit. Alternately, taste is the strategic high road, offering greater long-term value. To keep the messaging simple, we were hoping the granulated sugar reference wouldn't prove necessary, and that the 'taste' strategy performs on par with the dissolvability message.

Two goals for this week's test are:

1. Determine if the category liquid sugar needs a contextual reference to granulated sugar for clarity.
2. Determine the most effective features between dissolvability and taste.

A more in-depth explanation of these strategies and how they were arrived at can be found in the introduction article for this particular test.

Review of Key Metrics

We know what we'd like to find out. The question is: Which metrics will best clarify the issues for us? This week, we'll look at response rate data. The depth of visit data is still too sparse this week, but next week we'll cover that, as well.

As for levels of activity, ideally we try to get 100 actions within each data cell we're analyzing. This figure references Analysis of Variance (ANOVA) models that have been a common standard in online marketing for the past five years. The general practice is to go for 100 conversions in any given data cell. If there aren't at least 100 actions, we'll often go with as few as 50 -- go below that and the data gets unreliably sparse.

Here are the response metrics we'll be using for the analysis:

Visits per 1000 impressions: This metric gives us a more balanced view of the response rate than solely looking at the clickthrough rate. The validity of the clickthrough as a metric has been debated extensively. Without drudging it up, suffice it to say that actual site visits generated by clicks gives us a more qualified look at the numbers than just clicks without the ensuing visit. Furthermore, we balance the short-term nature of click activity with the longer-term view activity. Viewthroughs are people who see the ad, but find their way to the site in some way other than clicking on the ad.

Clickthrough: Per above, I'm generally pretty skeptical of this metric, although I use it occasionally as a second opinion on response rates.

Let's go to the charts and see how the data looks.

Charts and Analysis

Review Chart 1. Since we're focusing on response rates, this chart is based on the Visits per 1000 impressions metric.

Chart 1: Response Rate - Home Page Visits

Ad serving and reporting charts provided by Atlas DMT.

The different creatives/strategies are listed on the far left, under Creative Description. The bottom row, 'Strategy 3: Taste/Mix hybrid -- combined' is the total results of the Strategy 3 ad and its null version. Home page visits is actual visits to the home page, broken down by how the visitor arrived -- via a view or a click.

The impressions are very evenly distributed across the different strategies. Moreover, the four strategies are all reaching our goal of 100 conversions per data cell.

So what are the numbers saying?

First, let's evaluate the need for referencing granulated sugar in the ads. This would be by comparing the performance of Strategy 1 & 2 with the other two strategies, based on the Visits per 1000 impressions metric. This metric gives us accurate response rate data, as it keys on visits, and not just click activity. Furthermore, it brings viewthrough traffic into the picture.

The first thing we can see is that the difference among the granulated sugar strategies is noticeable, but not enormous. This isn't entirely surprising, since we're monitoring static ads with a conservative visual approach, which would tend to generate lower levels of activity.

Still, the metric Visits per 1000 impressions in Chart 1 is showing a 13 percent increase in performance going from the lowest to the highest strategy. That's not a significant increase, as various ads employing flash and dynamic graphics will often have 200 to 400 percent differences from the lowest to the highest. But we're testing strategic messages here, and those aren't going to drive the same type of results. And of course, even a 13 percent increase in performance in marketing is nothing to turn your nose up at.

We can view this from two angles. One is how Strategy 1 (taste with granulated reference) compares to Test 4 (only taste). They are running in an almost dead heat with more than 80 responses per strategy.

Secondly, we can look at Strategy 2 (dissolvability with granulated reference) and see how it's performing. It's at the lower end of the scale.

Chart 2: Response Rate - Clickthrough Rate

Chart 2 looks at the metrics based on clickthrough data. I mentioned I find this metric erratic, but at times it can help confirm other data sources by comparison. We can see that again, the granulated sugar references are at the lower end of our performance data.

Having two charts with significant data, neither showing an increase in performance as a result of the granulated sugar reference (and in fact, movement in the other direction) it seems the reference to granulated sugar is not needed for clarity.

Next, we'll look at the second goal of our testing to determine which product feature produces the best response rates -- Dissolvability or Taste.

Looking at the data in Chart 1, Strategy 2 is grounded in the dissolvability feature, and it's performing at the bottom of the pack, although the difference with its direct competitor, Strategy 1, is not significant.

The hybrid ad (Strategy 3) is doing well on both charts. This ad leads with the taste benefit, and states the reason as dissolvability. This is a convincing combination, providing both what appears to be the most compelling feature combined with logical proof of that claim.

One other consideration is that Strategy 4 (taste by itself) is also performing at the top, almost stride for stride with the taste/mix hybrid. This is good news, as this is a more simplified version of the taste strategy, and having that additional flexibility in the messaging could help us expand the creative messaging later.

Test Summary

Our goal was to determine if the idea of a liquid sugar needed a contextual reference to granulated sugar to add clarity. Based on these results, we don't believe that is necessary, as the granulated reference has performed at or near the bottom in all of our tests.

Furthermore, it looks like taste is emerging as the lead benefit, despite some initial research and information indicating that dissolvability might have more resonance. We'll take another look at these same metrics next week, when we'll have accumulated more data. We'll also examine how these different strategies were received by different target audiences.

More on that next Monday.

Doug Schumacher is the President of Basement, Inc.