As testing and optimizing becomes common practice among online marketers and merchants, the ugly underbelly of the testing process is beginning to expose itself. Despite the well-warranted enthusiasm, it is time to admit it:
Not all testing is good.
Yes, testing identifies opportunities for huge revenue improvements. Yes, testing helps you deliver more relevant and compelling experiences to your customers, increasing lifetime value and lowering acquisition costs. And yes, testing should still be incorporated into any online marketing program as a regular part of the marketing mix.
But that does not change the fact that it is actually possible for the process of testing to harm your relationships with loyal customers and hurt your bottom line.
So start testing, if you haven't already -- but when you do, be aware of these hidden pitfalls:
Pitfall #1: Surprising loyal customers
As I've said in the past, application forms, signups and shopping carts might be the best place to start testing because small percentage improvements can mean millions of dollars in revenue to medium- and large-scale sites.
But run a lousy test -- one that surprises or confuses customers who have already purchased from you -- and the negative effect from lowered conversions is just as great.
To avoid this pitfall, give visitors the opportunity to opt out of the test and return to the original, more familiar version. You might ask at the beginning of the form if they'd like to participate in a test of a new form. Or, you might clearly indicate within the form that it's a test, and allow confused customers to escape the new version and return to the old form.
Pitfall #2: Allowing an underperforming test to continue too long
Marketers occasionally find that some of their best ideas don't pull the results they had expected. When testing, if an idea falls flat and decreases conversions compared to the default, you could be losing money.
To avoid this pitfall, maintain real-time visibility into the performance of the test.
In a shopping cart test for a retail website, for example, we tested a variety of options for cross-sell items:
- Different product suggestions, including best sellers, most-viewed, and merchant picks
- Different price points
- Number of suggested items
To everyone’s surprise, it became clear after only six hours that more than half of the cross-sells were seriously underperforming the default branch.
In other words, those tests were actually decreasing conversions and losing money.
Fortunately, we changed a few things around and found a winning recipe (or combination of elements) that improved average order value by more than 16 percent.
With the ability to view your test results in real time, you can pause and evaluate if you see that the new version or versions significantly underperform compared to the default.
Pitfall #3: Running tests that will never reach a conclusion
You may have heard that successful, conclusive answers can be found after 20 conversions. In our experience, this rarely (if ever) happens.
More often, a test takes anywhere from 40 to 1,000 conversions per test branch to be conclusive.
This means that an eight-branch multivariate test requires 20 to 500 conversions per day to be able to conclude in two weeks.
The dual burdens of high expectations and limited resources can strain the patience of any marketer -- and a test that drags on forever may dampen the will to test. For a test to be useful, the results should be available in two weeks to a month.
To avoid this pitfall: Select pages that have adequate amounts of traffic.
Test elements that are likely to yield a significant signal (or indication, by either increased or decreased action, that the element is having some effect) to avoid inconclusive results.
Pitfall #4: Wasting time (and political capital)
Multi-variate testing allows you to test virtually an unlimited number of elements at the same time, which is an exciting ability for marketers eager to see which of their ideas will result in a lift.
The danger, however, lies in believing that anything and everything you test will produce a significant improvement. While small changes (font size, colors, minor copy changes) can occasionally make a big difference, more often they produce small results, or no results at all. Yet they take almost as much effort as bigger ones.
In fact, in a typical multivariate test, only about 25 percent of the elements tested will generate enough signal to produce a confidant prediction. The rest either do not have a strong enough impact, or the effect changes from day to day so much that the element never reaches statistical confidence.
To avoid this pitfall: Focus on big-impact elements such as price, product, promotional offer, total copy treatment, as well as the existence of elements on the page that are likely to make a big difference.
Then, select versions of these elements that are strongly different to avoid tests that show no impact.
Pitfall #5. Hurting your organic search ranking
When you run a test by splitting traffic and diverting it to two different pages, it confuses search engine spiders and can seriously harm your search engine rank.
To avoid this pitfall: Use tools that allow you to insert your various test elements into a single page without diverting traffic. For example, Offermatica stores test content options on its server, and then serves it to specially designated areas of a page (m-boxes).
When a shopper visits an Offermatica-enabled page, their browser communicates with our server to find out what content is to be displayed. If the visitor is determined to be a member of a targeted test group, they will get the content designated for that group for the duration of the test period, without their browser being diverted to another page.
The upshot of all these pitfalls is that, unless you truly understand testing at its deepest level, using inexpensive, "simple" tools for running sophisticated tests can actually hurt more than it will help.
But with a small amount of planning, using testing and optimization as critical marketing tools improves the return on your investments. And more importantly, it provides a steady set of conclusions to tune up your gut instincts.
Jamie Roche is a founder and president of Offermatica. Roche brings to Offermatica the experience of leading a visionary technology company from the dawn of the commercial Internet, through the bubble burst and out again. Offermatica, formerly Fort Point Partners, Inc. is an eight-year-old software company that provides hosted testing and optimization services to some of the largest retailers in the industry including Restoration Hardware and Joann.com.
Prior to Fort Point Partners, Roche ran Webfactory, a provider of Internet products and services to Yahoo, Netscape and other leading Internet companies at their formation. Roche also worked for KPMG Peat Marwick and SiliconGraphics. He is a graduate of Yale University.
