ellipsis flag icon-blogicon-check icon-comments icon-email icon-error icon-facebook icon-follow-comment icon-googleicon-hamburger icon-imedia-blog icon-imediaicon-instagramicon-left-arrow icon-linked-in icon-linked icon-linkedin icon-multi-page-view icon-person icon-print icon-right-arrow icon-save icon-searchicon-share-arrow icon-single-page-view icon-tag icon-twitter icon-unfollow icon-upload icon-valid icon-video-play icon-views icon-website icon-youtubelogo-imedia-white logo-imedia logo-mediaWhite review-star thumbs_down thumbs_up

4 critical steps for reliable email testing

4 critical steps for reliable email testing Tim Underwood

There is a simple way to explain the importance of a testing plan: It's the only way to make the leap from what you thought would work to what you know will work. The information that testing generates will help you understand what's best for your customers in real-life situations and gauge how they are interacting with your brand. This is the foundation every email marketer needs for success.

For example, if you have a sale on iWidgets, which subject line is going to appeal to the majority of people looking to upgrade? Be careful. You don't want the greatest number of people -- just the most qualified people. It's easy to get everyone to open -- just say "Free iWidgets!" But that's no good because when people open the email and see that iWidgets start at $29.95, you could have an angry mob on your hands.

You need to know what's effective and to be able to prove it. Here are four key steps to help you get there.

1. Plan ahead
First, compile a list of questions you need to answer. What elements go into a great subject line? Should I use images? What is the prime placement for my content? No doubt when you brainstorm around these questions, you will come up with great ideas, and some will really push the boundaries of your strategy.

In addition, don't waste time building a long-term testing plan that people will veto when it comes to executing it. For example, you may want to find out if your customers respond best to a percentage discount or a dollar value -- but will your legal department allow you to go to market with two such different propositions? You may want to test elements of the splash pages to which your emails link -- but do you have the resources required to develop multiple splash pages? If there are other people in the equation, get their buy-in early on.

Here is another key point: Only ask questions if you are prepared to act once you uncover the answers. You could ask customers if they prefer to hear from you every week or once a year. But if the latter comes back as the choice, are you really prepared to build a strategy around mailing once every 12 months? Probably not. So don't waste your resources.

Here is the most important factor: Test one variable at a time and make absolutely certain you can isolate that variable. (Of course, this doesn't apply in multivariate testing, which is designed to test multiple variables, but that's a more complicated topic for another article.)

Here's an example: Acme Industries wants to find out if offering 1,000 Acme Points for its new iWidget product will entice people to open the email. So the company decides to test two subject lines:

1,000 Acme Points with our new Super iWidget XS
The new Super iWidget XS -- just $29.95

In this scenario, it's impossible to isolate the reasons for either line's success because there are too many variations. We couldn't tell whether it was the 1,000 points that had the impact, or mentioning the price. The test is only reliable if the two variations are as similar as possible, like this:

1,000 Acme Points with our new Super iWidget XS
Our new Super iWidget XS

Here "1,000 Acme Points" is the isolated variable, so if the top subject line is the winner, we'll know why customers opened the offer. This is valuable information when it's time to release iWidget 2.0.

Next page >>

2. Execution
Now develop a list of variables to test. If you use an email service provider, it may be able to offer some valuable insight on quick wins you can initiate easily. Don't forget you will need to test everything twice to be certain of your results. A great strategy is to identify one stream of communications -- for example, your regular monthly newsletter -- and splitit into two versions. Whichever comes out on top, use that as the control for your next test. Testing should be a continuous activity, and by using this method you can ensure your marketing is always evolving.

Here are some thoughts on what you might want to start testing.

Subject lines: First thing first: You need people to open your emails if you want any hope of selling to them. You probably already suspect which aspects might resonate with your customer base -- try including price, sale deadlines, and unique variables such as loyalty points to see what piques their interest at a glance.

Content: You may always lead with a strong hero image at the top of your communications, but maybe it would be better to let the content float higher up. Is there value in segmenting based on geographic location and sending people content based on where they live? Perhaps your emails are very short and inspirational but would drive more qualified traffic if you included more detail.

Demographic segmentation: Do you always use bright pink buttons to compel people to your website? Perhaps male customers would respond better to a royal blue button, or maybe there is value to be gained from showing different products to different demographics.

3. Reporting
There are two key aspects to explore with regard to reporting: where to be looking, and how to find a winner.

You need to know which success metric is the most relevant to your test; there will always be a primary and a secondary. The subject line example we looked at in the introduction is the perfect example: Our primary success metric is the open rate, and this is really the only thing a subject line can hope to affect. Pay close attention to your unique click-through rate as a percentage of open rate, not just volume. If you have a great open rate but an awful unique click-through rate, you will know that your subject line is attracting the wrong type of people.

Content is a little more complex. You may develop an email that extols perfectly the virtue of a new product, proven by a record unique click-through rate. If you then find that these clicks are not converting to sales, something on the web page must have put them off, right? It's easy to blame the web team, but you might want to ask yourself some tough questions too. For example, if you got this great unique click-through rate by omitting the price from your creative, then your results are of very limited use.

Both of these examples consider the primary metric as being the action closest to the variable, and the secondary metric as the next action after that. This is really as much as you can hope to have influence over; any action further removed from your variable has been subject to too many other factors. You can't really attribute high conversion rates to a great subject line.

Lastly, it's essential to make sure your answers are statistically significant. Even two equally sized random splits will give slightly different response rates, so it's not enough to look at the highest value and declare it the winner. You will need to do some calculations to make this decision.

4. Take action
Testing initiatives can become very involved, so don't lose sight of the goal. Once you've got your first wave of results, make sure you share them with everyone involved and take action on them. But the job doesn't end there. Monitor your results to ensure your new best practices stay fresh and are still producing the desired results. Again, a good testing plan never ends. Your customer base never stops evolving and neither should your marketing plan.

Tim Underwood is an account manager at e-Dialog.

On Twitter? Follow iMedia Connection at @iMediaTweet.

Assist in both planning activities and the production of products and services, as related to Media Resources. Provide guidance to client teams on effectiveness, cost, schedule and execution of products and services. Creates protocol for...

View full biography


to leave comments.