Doug Schumacher recaps how the ads fared across different content channels.

Campaign Details: Client: Sugarshots, Inc. Agency: Basement, Inc. Ad Network: 24/7 Real Media Ad Serving + Tracking: Atlas DMT Site Analytics: Think Metrics   Useful links: Recap of past tests  Metrics overview  View entire campaign

After several tests, this is what we’ve been able to assess so far. The category of liquid sugar doesn’t need a contextual reference to granulated sugar for target audience understanding. And Taste is the primary benefit, as opposed to dissolvability. For more on the thinking and analysis behind those discoveries, refer to the Sugarshots archives.

This Week’s Test

In last week’s article I mentioned that our testing structure was the same for weeks one and two, and that I’d confirm the findings reported last week, based on the additional data gathered over the second week of testing.

The findings do seem to hold true across the second week (refer to Chart 1a). The first metric I’ll cover is click visits to the home page. We’re still seeing Ad 3 (Taste/Dissolvability) performing at the top, although there’s not an enormous difference between it and Ad 4, Taste Better. I also want to note I’m analyzing by the click visits column because at these small numbers, view data might skew the overall read. This excludes the view visits metric, which at these small numbers could create a less accurate influence on the overall read. So looking at the click visits column, we see that Ad 3 and Ad 1 are both leading by a small margin.

Our original hypothesis was that a granulated reference wasn’t necessary; taste as the preferred benefit seems to hold true. We’re thus proceeding with taste as the lead strategy, with a possible reference to the dissolvability, if we feel it adds relevance to the taste benefit. Actual purchase data is still too marginal to use for analysis.

Chart 1: Response rates for different ads on Home Page

(Ad serving and reporting analytics provided by Atlas DMT.)

A quick note about Ad 3 and the Null ad (See Chart 1b). This is the process of taking one ad and running an identical version of it against all ads in all the same placements. This enables us to gauge the variance of the data. In this case, Ad 3 and its null version pulled 60 and 72 home page click visits respectively. That’s a reasonable amount of variance. We’ve also seen the two versions of Ad 3 pull closer over the two weeks as impressions built, which is what we’d expect to see.

Chart 1b: Null ad test

Let’s move on to the target audience analysis.

Our goal is to analyze how our campaign performs across different target audiences, segmented by both content channels and by the behavioral targeting capabilities of our ad network, 24/7 Real Media, and their behavioral targeting tool, OnTarget.

We selected four content channels: Entertainment, Health, Men’s Interest and Women’s Interest. Additionally, we have three behavioral targeting overlays on top of each of those content channels: Health, Women’s Interest, and Static (our constant, with no behavioral targeting). For more on the strategy behind the role of behavioral targeting, refer to the article earlier this week.

Charts and Analysis

As for the key performance indicators, we’ll be using the same metrics as last week, with the response data based around Click and View Visits, which I often quote in its percentage form as Visits per 1000 impressions (V/1000i), for comparative purposes.

This week we have multiple creatives across four content channels, further subdivided by three behavioral targeting groups. This type of analysis typically involves a lot of discussion and explanation, so I’ll try and go through it as clearly as possible.

Let’s begin by reviewing the different content channels in Chart 2. Since we haven’t obtained a high number of impressions in the Health and Men’s Interest behavioral groups, we’ll need to consider the variance which that could create.

It’s evident that the Entertainment channel is performing well in the behavioral target groups. Comparing that content channel to the Women’s Interest channel, which also has good behavioral target impression levels, we see that Entertainment is clearly outperforming it.

In looking at V/1000i for Men’s Interest-Static and Women’s Interest-Static (Static means no behavioral targeting), there’s little difference in response rates. This implies there’s no gender skew for this product.

Within the Entertainment content channel, there’s a good distribution of impressions, meaning statistical significance, across the three behavioral targeting groups. The V/1000i column indicates both the behavioral targeting groups Health and Women’s Interest have outperformed the Static group. Further, going from the Static to the Health Behavioral group produced a performance increase of 262 percent in terms of traffic driven. The increase from Static to Women’s Interest is almost as dramatic.

In the Health Channel, there weren’t enough impressions in either of the behavioral target groups to draw any real conclusions. Interestingly, the Static Behavioral group within the Health channel did considerably better than the Static Behavioral group in Entertainment. We’ll be closely monitoring the Health channel as the impressions build within the Behavioral groups to see if their performance mirrors that of the Entertainment channel’s.

Chart 2: Analysis by Content Channel by Behavioral Target

In Chart 3, let’s quickly review the level of traffic to the Product pages. This metric is a way of qualifying the traffic, or making sure that the responses we’re driving are legitimate. Typically, the view visits cover that need, but given that our view traffic is very low (probably because the ads are static and intentionally bland), we need to get a second opinion.

The data is a little thin, but we do see a confirmation of the general trends with the response data; home page visits. The Channels that we see driving the best response rates are also driving the deepest interest by visitors.

Chart 3: Product Page Visits by Content Channel

Now consider how the data looks when sliced by behavioral target group, in Chart 4.

Returning to our V/1000i metric, we can see that all three behavioral target groups reached good response levels, and that the Health and Women’s Interest groups outperformed the Static group. Specifically, there was a 32 percent increase in performance between the Static group and the Health behavioral target.

Additionally, while there are a few Channels that didn’t obtain high impression counts, I feel the overall results are balanced by the fact that the behavioral target groups had lower impressions in Health, which performed at the high end, and Men’s Interest, which performed at the low end.

Chart 4: Response rates by Behavioral Target

It’s interesting to note that the Women’s Interest content channel is the lowest performing content channel, yet the Women’s Interest behavioral target is showing a strong increase in performance above the Static group. This isn’t unusual, and one reason may be the immersive quality of the content. Ari Bluman of 24/7 Real Media explains that if a given content channel has a strong immersive quality, it can be difficult to pull someone out of that mindset when they’re actually on that type of site. However, when they’re visiting a site they’re less involved with content-wise, they may be more receptive to ads pulling at their attention. And that’s where behavioral targeting comes in.

Test Summary

Clearly, the Entertainment and Health Channels are performing well, so it makes sense to concentrate future impressions in those areas. Additionally, there doesn’t seem to be a strong trend towards gender preference, further supporting dropping the Men’s Interest and Women’s Interest channels.

It’s interesting to note that the Health behavioral target within the Entertainment content channel drove some of the highest response rates. It makes sense that the Health behavior would tie to a food product with health overtones. And as I’ve previously mentioned, Basement and the Sugarshots team feel the product has a lot of potential lifestyle appeal. Strong performance within the Entertainment category would support this approach to the advertising.

As I mentioned earlier in this article, we’ll also be proceeding with the Taste strategy as the lead benefit without a contextual reference to granulated sugar.

Lastly, the performance of the individual creatives within the categories and within the behavioral groups also matters. The sheer number of data cells in this test makes it difficult to achieve enough activity in the more granular cells for accurate analysis. At this point, I’ve taken a general look at the creative data that’s significant, and can say that it’s surprisingly consistent with the high-level findings.

Going forward, we’ll continue to test across the Health and Entertainment content channels, as well as the Static, Health, and Women’s Interest behavioral groups within each channel. And since we’re employing fewer creatives and fewer content channels, there will be more impressions to go around.

Next week’s test is going to be considerably simpler. We’ll be trying to determine whether a package shot in the ad or a product usage shot drives better performance. Tune in Monday for the strategic thinking behind the test.

Doug Schumacher is the President of Basement, Inc.