Sugarshots Results: The Optimal Ad Unit

 

Useful links:

Recap of past tests 

Metrics overview 
View entire campaign 

 

The test objective is to determine the difference, if any, between two media units in terms of response rate and site activity, and to analyze them from a cost-value perspective. To view the introduction to this test, click here.

It’s difficult to assign value to space. Whether its office space, living space, or ad space, it usually involves multiple variables that aren’t easily compared.

However, the relative value of different ad units is what we set out to determine in this week’s test. Going in, we know that different media units generate different response rates. Obviously, it didn’t take online testing to figure that out. However, with different units comes different pricing, and the possibility of accomplishing different tasks, like branding or direct response. So the question we’re asking is, How differently do these units perform, and can we determine a ballpark relative value to that difference.

Testing Construct

The structure of this test is pretty simple: Run one creative message in two different sizes (300x250 and 728x90). Then run both units across parallel media plans, and evaluate performance.

The metrics of choice for this test is click-visits per 1000 impressions as well as page views and purchase page visits. We’re focusing on the click-visit activity data because we don’t have equal impressions delivered for the two different sizes. This ensures a more consistent, accurate analysis.

Given that we have multiple impressions per person, and the same person could easily see both ads, we’re excluding viewthrough data from this test, as well as the standard click and clickrate figures, which I’ll cover shortly. Thus, we will look at how many people visited the site from a click for every 1000 impressions the ad was shown.

Test Results

Chart 1 shows the response rate performance of the two ads. Looking at the click-visits per 1000 impressions column for each, we can see that the 300x250 has performed almost 90 percent better than the 728x90 ad. This level of difference easily places the data in a 99 percent confidence interval, which is a widely-accepted confidence level for most surveys.

Chart 1: Click-visit Response Rates

So why are we using the click-visit metric versus just clicks? Compare the number of clicks with the number of click-visits and you’ll notice quite a discrepancy. This is due to several things, according to a recent study by Atlas, our ad serving and tracking & reporting technology provider for this campaign. People often click on ads multiple times, for various reasons. The clicks metric records each click separately, whereas the click-visits metric attributes only one click-visit for each person who comes to the landing page (the home page in this case) via a click. That’s why the click-visit figures are considerably lower. The difference, called click-drop, varies, but needs to be taken into consideration when evaluating a campaign’s performance.

Moreover, looking at the clickthrough rate alone, it seems as though the two are quite similar when in fact, there’s a much wider gulf between their performance levels. This discrepancy can skew analysis considerably.

Now let’s look at how this tracks by target audience, as broken down via 24/7 RealMedia’s content channels and behavioral targeting. Looking at the click-visits per 1000 impressions column on the far right, we see the trend for the 300x250 banners is holding strong.

Chart 2: Response by Content Channel and Behavioral Target

With such a dramatic difference between these two media units, we’re now curious whether or not the difference extends beyond response rate data.

In print, it’s well-known that a larger size ad in the same publication increases post-impression brand awareness and recall scores. In other words, the response is not immediate, but rather involves longer term memory. Of course, it’s probably because the bigger an ad is, the more likely someone is to see it, and thus remember it, versus any sort of prestige that may come with the increase in size.

However, I think there’s at least a thin argument that media units of different sizes could have a different user experience. I know that 468x60’s just annoy me these days. Of course I’m jaded. Since we have the data, we might as well explore all the possible differences between these two media units.

Refer to Chart 3 for a depth of visit comparison. The average page views per visit are almost identical.

Chart 3: Depth of Site Visit

Looking at purchase intent, as determined by visits to the purchase page in Chart 4, we see the similarities again. There’s little difference between the two.

Chart 4: Purchase Intent

Ari Bluman of 24/7 Real Media informed me that it’s not uncommon to see 728x90s perform higher than 300x250s in certain areas, and I can imagine how that could happen on certain sites. For example, when the 728x90 is at the top of a home page, and the visitor’s attention is up around the navigation, they may be off to the next page without ever looking down in the main content area.

For this test, though, we’ve determined that the 300x250 pulled an 89 percent better response rate -- and at a significant confidence interval.

Our objective for this test was to determine whether or not there was a true difference in performance between the two media units, and to see if we could evaluate the difference on a cost-value basis. Although I’m not a media planner or buyer, I know a guy who plays one in real life, and he told me that the typical cost difference between 728x90s and 300x250s is around 10–15 percent.

In this case, spending 10 to 15 percent more for the 300x250 unit would generate an 89 percent lift in performance. Not a bad ROI.

There’s also another story in this data.

Consider the real estate each ad occupies on the page. The 300x250 takes up only  14.5 percent more space than the 728x90, yet generated 89 percent better performance. For a publisher wanting to maximize revenues, this could be key, as eventually, the price and effectiveness of a media unit should align.

So while the larger 300x250 unit should prove better for the publisher in the long run, it’s also better for the advertiser, and I’d even argue better for the reader. From a user-experience perspective, I’ll take a large ad over a page full of animated buttons blinking like a Christmas tree any day.

Doug Schumacher is the president of Basement, Inc.