Schumacher explains why Rasputin pulled in more click-visits than Lincoln, or the bottle itself.

 

Useful links:

Recap of past tests 

Metrics overview 
View entire campaign 

 

Tracking Report:

This week we move into the next phase in our series of online tests, past the foundational issues like message strategy and target audience to pit different creative head to head.

I mentioned in my setup article earlier in the week that our goal is to be able to apply these findings to mediums beyond online, including print, outdoor and point of purchase. To stay relevant to those mediums, we’ll continue to exclude animation from our ads.

The ads in this week’s test include the winner from the previous week’s test, which features an image of the Sugarshots bottle (I’ll refer to it as Bottle), as well as two new ads featuring images of Abe Lincoln and Rasputin (I’ll refer to these two ads collectively as the Historical Figures direction). For more on these creatives and the thinking behind them, refer to the last article.

Let’s take a look at the results from this week’s test.

Test Results

Chart 1 shows the click-visit response rates for the three ads. Rasputin did the best, with the previous winner, the Bottle ad, coming second, and Lincoln last. (For more on why we use metrics based on the Click-visit response rate, and not the more common clickthrough rate, review our metrics explanation. The difference in performance between Rasputin and Bottle is 2.7 percent, which isn’t much. I’d certainly test these to see if this difference spreads over time, or even reverses.

It’s interesting to note that Rasputin and Lincoln performed so differently. I’ll expand on this in the Summary section.

Chart 1: Click-visit Response Rates

Now let’s take a look at Chart 2 for creative performance across the individual media placements. We can see it is still quite consistent with the overall findings, adding credibility to the data.

As previously mentioned, we typically aim for at least 100 actions per each cell we’re trying to analyze. Not reaching 100 simply means we should expect more variance in the data, which should smooth out with additional impressions and responses.

Chart 2: Response by Content Channel and Behavioral Target

We know from experience that more qualified traffic tends to visit more site pages than less qualified traffic. This is often borne out in analysis of search-generated traffic versus banner-driven traffic. For our test, there’s a distinct difference in the amount of information our various ads deliver. Compared to Rasputin or Lincoln, Bottle states liquid sugar up front, gives additional information about its dissolvability, and even shows the product. The question is thus, will the traffic generated by a more information-based ad be more qualified, because like search traffic, it’s a seemingly more relevant frame of mind for the content on the website?

An initial concern regarding the Historical Figures direction was the possibility of the campaign generating high response rates, but not delivering on the backend, as people visited out of curiosity, without a real interest in liquid sugar.

To gauge whether or not this is happening, we compared the average number of pages visited for traffic coming from each of the ads. We surmised that if someone responded due to a fascination or curiosity about Rasputin and didn’t give a click about liquid sugar, they’d be likely to hit the home page and bounce off immediately, or at least more quickly than someone with a higher level of interest in liquid sugar. 

Click Page Views per Visit is the average number of page views for each visitor that arrived by clicking. Chart 3 indicated the Click Page Views per Visit data is closely following the trend we’re seeing in the response data. It implies that the traffic coming from the Rasputin ad is viewing roughly the same number of page views as the visitors from the Bottle ad. The implication of this is that the traffic Rasputin drove was just as interested in liquid sugar as the traffic from Bottle. 

Chart 3: Depth of Site Visit

In Chart 4, we’ll look at purchase intent, as defined by the number of click visitors reaching the Purchase page to gauge real interest in liquid sugar. Interestingly, Lincoln performed the best here, although the data is pretty close, and the low cell count makes any variance look big proportionally.

Chart 4: Purchase Intent

Summary

I noted in the previous article that Phase Three of our testing, the Emotional Drivers, is where we start loosening our tight grip on controlling the variables among the different ads. Comparing Bottle to Lincoln and Rasputin, there are a number of attributes that might explain a difference in performance.

One way to help us analyze what’s working is to test different creatives along a similar strategy. That’s why we ran both the Lincoln and the Rasputin ads. Looking at the results, it might be surprising that Rasputin and Lincoln are so far apart. However, creative performance often breaks along very different lines than what might seem logical.

Why didn’t Lincoln do so well, especially compared to Rasputin?

We can clearly pinpoint the difference between the two ads. And comparing the two visuals, we could ascertain that perhaps featuring Lincoln in any ad is a little ho-hum. Lincoln’s figure is a common theme in logos, financial marketing, and President’s day sale ads, to name but a few examples. So perhaps featuring him here was just adding one more Lincoln ad to that mix.

To explain further, we can view the ad performance in light of the hierarchy of effects model, which outlines stages that an ad moves a viewer through. It’s also called A.I.D.A., with the letters standing for the different stages: Attention, Interest, Desire, Action.

Thus if Lincoln’s image is de rigueur, then that ad fails at the first stage, Attention. Furthermore, if Lincoln’s image is indeed embedded in our collective American brains, this might even be a situation where even though the Lincoln image may pop off the page more than the bottle image, the Lincoln image is so stale that readers simply can’t be bothered to give interest to yet another piece of corporate communications starring Honest Abe.

Rasputin, on the other hand, is certainly a less common subject. He cuts an intriguing, if not good looking, figure.

From my experience running tests like this, I’ve always found that creative performance doesn’t necessarily follow strategic paths.

With Rasputin and Bottle testing very closely, we’re left with several options or decisions to be made. While the test was conclusive from a statistical significance standpoint, I’d continue to run these two side by side and monitor them to at least the point where most of the individual cells in the content channel chart hit or get close to 100 actions.

I would also keep Bottle and Rasputin running, but bring in other historical figures, both known and obscure, as well as ones that break along different lines, real or imaginary, male or female, and possibly heroic versus disdained. (If they were alive today, it’s difficult to imagine someone with Rasputin’s reputation being a paid endorser for anything. Alternately, Lincoln would seem to have his pick of endorsement deals.)

But even if they continue their close run, I have to say that between the two, I’d opt for the Rasputin ad. Perhaps this is my traditional advertising background coming out, but I think creative along the lines of Rasputin helps build more brand personality. Certainly when compared to a standard looking product shot. It’s more unusual graphically, and that can provide images to play off in site content, other marketing materials, and even promotional themes.

So at this point, I’m sending Lincoln back to his grave and keeping Rasputin up in the land of the living and the gainfully employed.

Doug Schumacher is the president of Basement, Inc.