Sugarshots Results: Flash's 2nd Chance

Recap of past tests 

Metrics overview 
View entire campaign 

A few weeks ago, we ran a test in which we compared an animated ad to a static ad. Unlike many similar tests I've run, the data showed the static ad clearly outperformed the animated ad. Although that seemed a little surprising, the area of animation is extremely broad, with numerous ways to approach animation. With this in mind, as well as the belief that testing is an iterative process, we decided to test a second approach to animation.

In our previous article we took a close look at the first style of animation, and tried to assess why it may not have worked relative to the static ad. We also presented the second round of animation, stating why we felt it could bring a performance increase over the first approach.

With that in mind, we'll look at the numbers and see how the new animated ad fared.

Test Results

Animation really focuses on the attention-getting qualities of an ad. So our primary focus with this test is the response metric click-visits per 1000 impressions. For an explanation of why we use this metric as well as our other metrics of choice in these tests, please refer to Metrics Overview link to the right.

Chart A below shows that once again, the static ad outperformed the animated ad. The difference this time is much smaller than our last test. However, the static ad is still generating a 56 percent increase in performance over the animated version, based on our key metric, click-visits per 1000 visits.

Chart A: Response Rates

This is certainly not an insignificant difference. As for the legitimacy of the findings, we're well within the 99 percent confidence interval, which is the generally-accepted level of reliability for most polls and surveys.

Now let's get a more granular view of the data, reviewing performance across the content channel and behavioral target subsets of the media plan.

Chart B shows how the performance of static compares to animated across all media placements. (Note that we excluded any media placements that didn't have at least 80,000 impressions, due to lack of data.) The column Click V/1000 is the column to focus on.

Chart B: Response Rates by Media Placement

By comparing the static creative with the animated for each media placement, we can see that the data tracks consistently across the various placements. If performance varied greatly by placement, we could assume that there was a bit of noise in the data. However, having the performance this consistent across so many different media placements gives us a lot more confidence in the results.

Lastly, as a qualifier for this, we'll consider the page views per visit for each ad. Chart C shows the Animated ad to have a slightly higher page view rate, the difference is not really that significant. If these two were much more out of alignment, I may feel there's something unusual happening behind post-click or post-impression, but that's not the case here.

Chart C: Page Views Per Visit

Summary

To reiterate what I said at the end of our last animated versus static test, the data surprised me based on past experiences running similar tests for a range of products and services. There are also a couple of things to consider with these results:

First, there are a number of variables that can make the results for one test scenario different from another. These include creative type, publisher or network, media unit, product, and target audience. So while one type of creative execution may see a good lift by implementing animation, another approach may work best in a simpler form.

Second, I've seen a number of still ads that grabbed my attention. In particular, I remember the "I Heart Huckabees" campaign. Many entertainment ads use extensive animation. The "I Heart Huckabees" ads simply featured close-up still photos. (Granted, they were photos of great looking actors, which can't hurt.)

Another consideration is time. Animation is about standing out and separating your message from the surrounding noise. What stands out one minute may not the next. When most ads were in black and white, a color photo stood out. Several decades later, black and white photos were used to stand out in a color world.

With the growing use of Flash and broadband adoption, animation on the web is certainly not as unique as when Flash first came on the scene. With that being the case, it would track that animation doesn't stand out as much, and thus would be as strong of a differentiator.

Whatever the reason for the outcome of this particular test, I'm certainly not ready to drop animation as a common practice. I will, however, run this type of test more frequently for a range of other client products.

Lastly, if anyone out there is running similar tests and wouldn't mind sharing the information with me, I'd be curious to hear how other tests are panning out. It's open source marketing; the more the merrier.

Doug Schumacher is the president of Basement, Inc.