Looking Back at Sugarshots

If you're a regular iMedia Connection reader, then you've joined us over the last three months as we've launched an exciting new kind of content: open source marketing.

Doug Schumacher, president of interactive agency Basement, Inc., has been leading the charge on openly testing the creative for his brand client, Sugarshots, getting commentary from other iMedia contributors, and generally letting our readers peek under the hood at how he does his job.

iMedia's mission is to advance the business of interactive media and marketing by serving as the primary conduit between buyers and sellers. And, to inspire marketers of all types to explore and embrace interactive marketing strategies. Open source marketing enables that mission, and in the coming months you can look forward to more open source case studies: not restricted to creative tests but embracing the full range of interactive marketing.

Last week, we wrapped up the final Sugarshots test. So today we're taking a step back to see what we've learned from Sugarshots and this first exercise in open source marketing. iMedia's associate editor Emma Brownell, who has been the organizer and air-traffic controller for our side of the campaign, joined Doug and me for the chat that follows.

But before we get there, I'd like to thank Doug Schumacher for bravely taking this task upon himself, and I'd also like to thank our industry contributors: Amy Auerbach, Ari Bluman, Reid Carr, Dave Chase, Brandt Dainow, Eric Porres, Jamie Roche, Matt Schow, Doug Weaver and Doug Wintz.

Finally, a great big thank you to our generous donors: 24/7 Real Media provided an astonishing amount of ad inventory that made the tests possible. Atlas DMT enabled our ad serving and tracking, and ThinkMetrics provided site analytics at the drop of a hat, not to mention regular iMedia contributor and ThinkMetrics CEO Brandt Dainow's insightful commentary.

(And if you're new to Sugarshots, then please take a moment to bookmark the hub page, which you'll find useful.)

Brad Berens: From the 30,000 foot level, if a visitor to iMedia Connection were coming to the Sugarshots hub page for the first time, and had the entire campaign to look at, what would you want them to look for going in?

Doug Schumacher: Online testing has broad applications, and can mean different things to different people. The intro article provides a level-set on the objectives these tests were built around. So that article would be a logical starting point for understanding the overall perspective. From there, I'd suggest reviewing the 'Recap of Pasts Tests' popup window. It's a snapshot of each test and its objective.

One note I should add is that this particular series of tests is more a showcase of online testing capabilities and methods than an absolute approach to launching this or any other product. However, I'd recommend a number of these tests for a company in Sugarshots' position. In particular, the tests on strategy, emotional drivers, and target audience. Others, like animation versus static, are really outside a company like Sugarshots' immediate interests.

Emma Brownell: The fact that the static ad outperformed its animated counterpart was a surprise. Throughout the campaign there have been unexpected, and expected, results to the various tests. Looking back, do you feel that you have some baseline information that you didn't before on how to market a niche CPG? What is it?

Schumacher: CPGs run a broad gamut in terms of product types, so there's going to be a lot of variance as to how each is approached. The process we laid out for Sugarshots provided several key benchmarks for strategy, target audience and creative, which are baselines that will help any startup coming out of the blocks.

Niche products in any category provide the challenge of trying to gauge the target audience's familiarity with the category. Naturally that varies considerably by category and brand.

What we found with Sugarshots is that although liquid sugar is fairly uncommon in the States, people were able to grasp the concept, even in the compressed format of an online ad. That finding enabled us to take a higher road for the value proposition, and gave more latitude to the creative executions. Later tests pointed back to the initial strategic tests and confirmed their results.

Berens: In his article on open source marketing, Dave Chase argued that more marketers should be "'opening the kimono' on portions of their marketing plan," but with Sugarshots it wasn't his kimono! It was brave of you, Basement and Sugarshots to do this. Did you ever have any regrets? Did the extra scrutiny of the iMedia Connection readership make you more focused? Nervous? Were there any moments when you thought, "Oops, if only I hadn't done this..." Conversely, what's the best unexpected insight that you got from one of our contributors? Did the open conversation add anything to your practice that you'll carry with you?

Schumacher: The thing I was most anxious about was writing 26 articles in three months. They have so many more words than ads.

The open source marketing concept reminded me of my first exposure to campaign metrics. Coming from the traditional side, getting such immediate and seemingly absolute feedback was a little harrowing. Calluses build up quickly, though.

At the other end of the scale, putting our thinking out there and seeing the response is one of the more interesting aspects of this business, in my opinion -- be it an ad, article, a methodology, or whatever. I'm not sure whether I should be surprised or not, but I had several thoroughly enjoyable exchanges with readers.

There are so many different backgrounds coming together in online marketing right now, and getting a look at how different people approach the same problem is fascinating.

It was the same with the iMedia contributors. We had a wide span of backgrounds participating in the campaign, which made for great diversity in the topics. Some were tactical, like Jamie Roach of Offermatica. That was one of my favorite tests, partly because of the remarkable technology they've developed. Previous ad tests revealed that more entertaining, lifestyle approaches were working better than product-centric ads. With Offermatica, we then tested for the same drivers but focusing on site content and utilizing completely different methods of evaluation. Test results showed the same fundamental themes working in both instances.

In contrast to tactical observations, Dave Chase and Doug Weaver each brought a global view that I felt gave valuable perspective to what iMedia case studies are all about. They both reinforced the idea that open source marketing is not going to be a perfect world: there are going to be missteps, there will be differing opinions, and so on. But the end result will more than justify the process.

I read a quote recently by Barry Diller on his activities with IAC, "When you are trying to do what we're doing, if you're not wrong about some things, you're really wrong." That sums it up pretty well, I think.

As for what could have been better, it was primarily about time and proportion. There were several tests that I would have liked to have spent a month constructing instead of a week, and dedicated twice as many impressions to. The target audience and the tests with Offermatica are examples, simply because I felt we were only scratching the surface in terms of the potential.

As for the Sugarshots commentators and what I'll take away going forward, every one of them had insights and views that have stuck with me. It goes back to the range of perspectives we had from the contributors. We had a lot of industry expertise weighing in.

There were several unexpected insights throughout the tests. The initial strategic tests were particularly noteworthy, primarily because they brought a fresh perspective to how Sugarshots was approaching their marketing strategy. They had initially focused on the dissolvability of the product. What several tests pointed to, though, was more of a taste benefit. In retrospect, I think that makes sense. Taste delivers a higher-level value, and is more about consumer benefit than product attribute.

I thought the test between the 300x250 and 728x90 media units was one of the most revealing, although it was something I've also experienced in our own campaigns. What was significant was that a media unit only 14 percent larger delivered an 89 percent increase in performance with about a 12 percent increase in cost.

And of course I got a certain satisfaction seeing the ad without a button so decisively outperform the ad with a button. I expected a difference, but nothing that extreme.

Brownell: Do you think certain things worked or didn't specifically because this product is a new product in a new sub-category? Can you speculate about what we might have learned had this product been competing in an already-established market?

Schumacher: The new product category issue was definitely the source for a lot of questions going into this series of tests. It meant that topics that may have been considered closed cases in a different situation were wide open. Target audience and value proposition are good examples. So, yes, in a more mature product scenario you could circumvent those tests and move directly into the emotional drivers, looking at creative executions or promotional directions.

That said -- if you have a mature product or category and you reconsider target audience and strategic positioning, new approaches in those areas could lead to expansion into new markets. I've used the term "creative" a lot in reference to creating the ads, but Open Source Marketing has revealed yet again that creative thinking is a key element across the marketing mix, not just key to one department within an agency.

Brad Berens is iMedia's executive editor.