Doug Schumacher examines how the image of Sugarshots packaging stood up against a picture of iced tea. 

Campaign Details: Client: Sugarshots, Inc. Agency: Basement, Inc. Ad Network: 24/7 Real Media Ad Serving + Tracking: Atlas DMT Site Analytics: Think Metrics   View entire campaign

As noted in my last article, whether or not to feature the product as the primary visual in an ad seems to be one of the more commonly debated creative issues. That, and logo size. So for this week's test, we placed a product shot and an alternate image head to head.

For the alternate image, we selected a straightforward table-top photo of a glass of iced tea, which is how a lot of sweeteners, natural and artificial, are used this time of year. The iced tea image also works without needing to change the headline, and thus made this a pure comparison between visuals. Isolating the visual means that if there is a difference, we'll know exactly what it's due to. Incidentally, sweetener advertising has often featured appealing food shots as the dominant visual, clearly aiming to add credibility to their claims of quality taste.

To read about the thinking behind these ads or to view the ads, please refer to Monday's article.

With online testing, sometimes the data run neck and neck, and sometimes it's a landslide. What we're seeing this week is the latter.

Charts & Analysis

The metric we'll focus on in this analysis is click-visits to the site. This is different from simple banner clicks, in that it represents actual site visits, and is more representative of the impact of the ad.

Please refer to Chart 1 for overall response data for this week's test. The click-visits for the product image ad are about 66 percent higher than the iced tea glass banner -- definitely a significant difference.

Chart 1: Home Page Visits

Driving traffic is only one aspect of online advertising, though. Even "punch the monkey" drove clicks. We need to qualify the interest of visitors to the site. We'll look at the page views that each ad generated as a way of gauging real product interest.

Looking at the page views data in Chart 2, we can again see that the product image ad is not only driving better initial response rates, but that the level of interest has actually increased for the product image. In this case, it has generated 68 percent more page views than the other ad. Had the product image traffic shown little interest in the site beyond the home page, we could have likely concluded that it was great at generating a knee-jerk response, but that the visitors weren't genuinely interested in the product offering. That's not the case here, however.

Chart 2: Total Page Views

Now, page views isn't the end-all metric. I think it's particularly relevant to a new product, especially when the site is small, informative and easy to navigate. But it's good to evaluate campaigns from an ROI perspective whenever possible.

As previously mentioned, given the product launch and new category conditions of this campaign, combined with the only real revenue action being online purchase, basing campaign performance on actual sales is not realistic. However, we do want to find a way to gauge the level of purchase interest. We'll thus look to purchase page visits as a strong indicator of purchase intent. True, even window shoppers could click this to check pricing, even if they're not considering an online purchase, but not everyone who visited the site went to the purchase page, so that page represents the last stop on the way to the cashier.

Chart 3 shows the click traffic levels to the home page, the purchase page, and the percentage of home page visitors that went to the purchase page.

Chart 3: Purchase Page Compared to Home Page Visits

Clearly, the product image ad is still holding strong, having driven 142 percent more purchase page visits than the iced tea ad. Adding further support, a higher percent of the product image audience visited the purchase page than did the iced tea image audience.

Having performance support one ad so consistently across so many data points makes a convincing case for the product image ad. However, there's one final way we want to slice this week's data.

Chart 4 shows the performance for each ad relative to the target audience segments in our media plan. (We've narrowed this down to the content channels and behavioral groups in which we've reached statistical significance.)

Chart 4: Content Channels and Behavioral Groups

A quick scan shows the trend is continuing across the content channels and behavioral groups in terms of page visit, page view and purchase page data. The data gets a little thin down at the purchase page level when broken down by target audience, but even at this early stage, a clear pattern seems to be emerging.

Test Summary

Our goal for this week's test was to answer a question about creative execution that repeatedly resurfaces:
Which will be the more effective visual, a product shot or an alternate image based around product usage? In this case, the product shot has shown a clear lead across a number of performance check points: response rate, product interest and purchase intent.

Some of the specific data findings include:

• Response rates: The product image ad drove 66 percent more site visits than the iced tea ad
• Depth of visit: The product image generated 68 percent more page views
• Purchase intent: The product image generated 142 percent more visits to the purchase page
• Visit to purchase page conversion rate: The product image ad also drove 23 percent of its site visitors to the purchase page, versus 16 percent for the iced tea ad
• Overall, 22 percent of the site visitors went to the purchase page (the above chart says 20 percent, but I'm counting view visits as well)
• Continuity across target audiences: We noticed the performance trends held true when broken down by our target audience segments

While the data in this test convincingly points to the product shot, I'd like to discuss a few important qualifiers.

First, the range of visuals that could be tested against the product shot is as broad as the imagination. We kept the two ads as similar as possible except for the image to build a baseline for further testing. The idea behind using something other than the product shot is to open up the creative possibilities on all fronts: visual, headline, layout, et cetera. We'll be exploring those options in upcoming tests.

Second, like most online tests, this one is highly situational. Even for other new products in the CPG category, your results may vary, as the lawyers say.

Third, test results can sometimes have a short shelf life. As the brand's awareness builds and the liquid sugar category becomes more familiar, an ad featuring a product image and the term 'liquid sugar' may not be enough to generate interest. Market conditions, the competitive landscape and consumer taste are constantly shifting. Ongoing testing can help you stay in sync.

Doug Schumacher is the President of Basement, Inc.