Sugarshots Results: Chic vs. Quirk

 Campaign Details:

 

Useful links:

Recap of past tests 

Metrics overview 
View entire campaign 

Tracking Report:

Standup comedians are well known for honing their craft on the road schlepping from one comedy club to the next. They'll try a joke out in Chicago, see how it plays out, make a few tweaks, and then try the revised version in St. Louis two nights later. What I find interesting is that they generally don't try a joke out, and then keep it as is or toss it out. More typical is that jokes and routines are crafted over time, as they test, try, and retest, to find just the right mix of timing, emphasis, and punch line.
I think in creative testing, it's easy to judge ads as pass/fail. We think of one ad as working, and another as not working. In reality, however, a few minor factors could significantly shift the performance one way or another. Going into this week's test results, therefore, keep in mind that this single test is likely not the end of the story. It may be that we should make our own tweaks and adjustments and then try the new material out in the next town.
With that, let's take a look at the results.

Test Results

We'll start by reviewing Chart 1, which tracks click visit response rates. In the past, we've been keying on the click visits data column for comparison, but this week, the impression counts for the two banners are slightly different, so we'll evaluate them with a relative performance metric, click visits per 1000 impressions. (Anyone who doesn't understand the metrics we're using, please consult our Metrics Overview.)
Immediately noticeable in Chart 1 is the fact that the two ads have the same click visits per 1000 impressions rate. The cell data is also well above our 100 actions target. This is quite surprising, given the significant differences in the creative. Additionally, both creatives are performing well, especially considering that there's no promotional offer in either message and we still haven't employed Flash animation. (Using Flash animation would pretty much wipe out any relevance in applying our test's findings to print or other static advertising vehicles.)
If the creatives weren't performing well, it would be easy to dismiss the results. Since that's not the case, we'll take a deeper look at the data.

Chart 1: Click-visit Response Rates

Ad Serving and Tracking provided by Atlas DMT

The page views per click visitor data in Chart 2 is where we see some difference between the two ads with Rasputin showing a 10 percent increase in performance over Café. This data means the traffic that came after clicking the Rasputin banner viewed on average of 10 percent more pages than traffic coming from the Café ad.

Chart 2: Page Views per Click Visitor

Taking this one more level, Chart 3 reports the purchase intent of the visitors coming from each banner. This is the percent of visitors from each banner who clicked on the purchase page, which is the only page featuring pricing information.

Here we see Rasputin leading by a considerable margin with 10.9 percent of the click visits to the purchase page, compared to only 4.76 percent for Café traffic. That's a performance lift of 129 percent, which given how much closer this metric is to purchase behavior, is certainly worth noting. 
Summary

The data for this week's test shows an interesting trend. While the response data between the two creatives is almost identical, the further we attempt to qualify the visitors, the more the data points towards Rasputin. In trying to explain this, let's examine the creative difference between the two ads.
From a messaging standpoint, Rasputin defines the product category as liquid sugar more clearly; it's right there in bold letters in the headline, though they both clarify the category in the tagline under the logo. To be sure, our feeling going in was that with little content in each ad, most visitors would read that tagline before clicking on the ad.
Of course, we try to assume as little as possible. If a number of the Café viewers didn't see the 'liquid cane sugar' tagline, then they would have clicked the banner out of curiosity. The tagline is their only clue that this is a liquid sugar product. People clicking ads due to curiosity aren't as qualified, and tend to not visit the site as deeply because they're just wondering what the ad is all about.
One asset the Café banner has going for it is it's much more in line with the website branding. Illustrations like the one in Café are featured throughout the Sugarshots site. Generally, a consistent brand experience between the ad and the website should improve depth of visit stats. Of course, the other creative variables may be offsetting the positive impact of Café's brand cohesion. 
In retrospect, I think it's easy to look at these two banners and think the results are obvious. However, keep in mind that we just tested Rasputin versus the Bottle ad. In that case, Bottle not only stated 'liquid sugar' more overtly, but also explained why liquid sugar tastes better, and even showed a picture of the product. Despite that, Rasputin performed equal to Bottle across the same metrics we're using in this test. In a situation like this, I like to continue pushing outward until we hit resistance, and then assess the results. Previously I've mentioned that I always try to simplify ads to their bare minimum. So when I see the term 'liquid sugar' in the tagline, I like to see if I can get away without repeating it elsewhere. It's about testing for boundaries regarding how little information we can deliver, and still be relevant. Perhaps we found that boundary with Café.
For the most part, online ads don't have the luxury of using body copy to offer additional explanation to a concept. Readers may visit the site for the second half of the equation, but if the tracking data indicates they're not really interested, then we need to reconsider the clarity of our initial messaging.
Carrying forward the results of this test, I would ascertain whether Café's site performance is due to differences in information or something else. To that end I would add a more overt liquid sugar reference to the Café concept. If that increases performance, then we can deduce that it was a lack of information. Secondly, I would change the Rasputin concept to say 'Rasputin on Sugarshots' omitting 'liquid sugar' from the tagline. If that drops the performance, we could deduce that one key to Rasputin's success is the liquid sugar reference.
One of the ongoing objectives throughout these tests has been to gain a clearer understanding of the consumer's familiarity and comprehension of the liquid sugar category. It's about delivering Sugarshots' message within the right context. It's also about knowing what information to include in the ad. Explain too little and they might not understand you. Explain too much and you could simply bore them. Either way, you risk miscommunication with your audience.
I feel there's good potential for Café to be brought up to speed with Rasputin; like new material in a comedian's routine, it may need a little tweaking. But with the right adjustments, it just might live to play prime time.
Doug Schumacher is the president of Basement, Inc.