Offermatica's Jamie Roche explains the differences between A/B testing and multivariate testing, and when to use each kind.

Successful marketing ideas spring from the minds of great marketers. Without the brainstorming, hard work, and experience of inspired marketing minds, no campaign would get off the ground.

But even the most experienced marketer knows the frustration of watching a killer campaign bomb. The idea was brilliant, the implementation flawless, and the ROI … nonexistent. That's because, in any campaign, so many elements exist that can affect a consumer's behavior that it's impossible to predict what the consumer's action will be.

Let me say that again: Impossible. Don't believe me? Consider this:

We recently ran a "marketing IQ" quiz for some of the most knowledgeable marketers in the e-retail space, showing four different "treatments" -- variations on copy and design -- of a single promotion page that PETCO ran as a test to see which was the most effective. We asked marketers to choose, based on their experience and gut instinct, which treatment converted the best.

Only 26 percent of those who took our quiz chose the correct treatment. (You can take the quiz here.)

That means 74 percent of marketers guessed incorrectly, in spite of their expertise and know-how.

"The biggest thing we learned is that your natural instinct isn't always right," says Heather Blank, Director of e-Commerce Marketing and Business Development for PETCO.com. "What we all thought would be the best discount wasn't necessarily the case."

Heather was sure that the biggest discount level at the lowest dollar threshold would result in the highest lift in total sales and margin -- but in reality, a different combination worked best. 

Simple, small changes, such as the wording of your call to action (example: "shop for ink" vs. "find your cartridge") can have a big revenue impact, but you can't know for certain until you test your ideas. Testing helps to separate great ideas that work from great ideas that don't work. When done right, testing also allows you to isolate which variables are helping, and which hurting, your conversion rates.

A/B Testing

When you create two versions of a single web page, divide visitors into two groups, and show each group a different page to see which converts better, that's an A/B or Split Run test. At its simplest level, it's like comparing Coke to Pepsi -- you test two completely different pages to see which treatment sparks the best response in visitors.

Another type of A/B test occurs when you experiment with changing one single variable -- a picture of a product on a model vs. a product against a white background.

By comparing conversions, average order value, form completion, or whatever task you're hoping visitors complete, you can tell -- scientifically and with no guesswork -- which version works better in the real world.

A/B testing is the easiest way to let your visitors tell you what changes affect their behavior. You can create an A/B test using simple JavaScript on your pages, or you can ask your IT person to set up routing between two versions of your application and use your site analytics package to tag the different versions. If these are not available to you, there are companies who can set up simple A/B tests for you for a fee. 

Be aware, however, of some serious limitations:

1. There are always costs involved in running an A/B test. You have to create two versions, set up your analytics to capture the results and spend time (or money) preparing your site to show different visitors different content. Unfortunately, more often than you expect, the new B version does not perform any better than the A. 

A/B testing is like drilling for oil. Not every test produces strong positive results. But every test costs money.

2. Even when your test shows a positive improvement, you won't know exactly which elements are helping and which are hurting. For example, if you test a new hero shot, you may learn that the new page with the hero shot improves conversions by 3 percent. What you won't learn is that another element, perhaps your call to action, is negatively affecting conversions. In other words, the new hero shot may have a +20 percent impact overall, but because of the negative impact of the call to action you're only seeing an insignificant (+3 percent) improvement.

If you knew that your call to action was weak, you could certainly fix it -- but with A/B testing, you may never find that out.

3. In order to have confidence in your A/B test results, you may need to test thousands of visitors (or anywhere from 50 to 200 orders) over a minimum period of two weeks. Imagine if you wanted to test four different product pictures, four headlines, four different navigation styles, and four different promotions. That's a total of 64 A/B tests to run -- which would take far too long and cost too much for all but the most committed marketing tester.

Multivariate Testing

Multivariate testing, on the other hand, allows you to test all of those elements mentioned above, and more, using only a small subset of recipes. In other words, it allows you to estimate the positive or negative effect of each single element, as well as the best combination of elements, even if the specific combination was never explicitly tested.

This means that you'll know why one change is better or worse than another, and how it interacts with other changes. You'll know that the new hero shot blows all other images out of the water, and you'll know that your call to action is weak. You'll discover that your new navigation box gets visitors to the correct product page more quickly than the old navigation, but only if you use headline B as opposed to headlines A, C, or D.

Multivariate tests can give you all this information much more quickly, and with less traffic and less effort, than other kinds of testing -- an important point particularly for niche marketers who typically deal with fewer overall visitors.

But here, too, there are drawbacks:

1. This scientific approach is difficult to build in-house. It requires long development time, committed IT staff, and the purchase of expensive software and/or hardware. And unless you have an IT staff that truly understands the science behind it, you can run into real troubles quickly.

2. You run the risk of drowning in the details. If you take into account every font and font size and font color, every minor variation in wording, every apparent difference in white space vs. the use of graphics, you can easily run up against a million different combinations. Unless you have a strong sense of what may or may not affect conversions, you can get so caught up in details that you miss the boat when it comes to implementing important changes.

Luckily for the marketer, a new generation of optimization services has arrived to meet these objections. Now you can find solutions that are hosted and administered specifically by and for the marketing user, that offer continuous testing, and that run on existing pages -- rather than using redirects -- to preserve search results (a key point, considering the amount of time marketers spend on enhancing organic listings).

Once marketers have a system for testing in place, they find they use it as an ongoing part of their marketing programs, to test and optimize existing pages, advertising and email campaigns, landing pages, and even entire user sessions for the best possible results.

You'll never have to "guess and hope for the best" again.

Jamie Roche is a founder and president of Offermatica. Roche brings to Offermatica the experience of leading a visionary technology company from the dawn of the commercial Internet, through the bubble burst and out again. Offermatica, formerly Fort Point Partners, Inc. is an eight-year-old software company that provides hosted testing and optimization services to some of the largest retailers in the industry including PETCO, Restoration Hardware and Joann.com.

Prior to Fort Point Partners, Roche ran Webfactory, a provider of Internet products and services to Yahoo, Netscape and other leading Internet companies at their formation. Roche also worked for KPMG Peat Marwick and SiliconGraphics. He is a graduate of Yale University.