ellipsis flag icon-blogicon-check icon-comments icon-email icon-error icon-facebook icon-follow-comment icon-googleicon-hamburger icon-imedia-blog icon-imediaicon-instagramicon-left-arrow icon-linked-in icon-linked icon-linkedin icon-multi-page-view icon-person icon-print icon-right-arrow icon-save icon-searchicon-share-arrow icon-single-page-view icon-tag icon-twitter icon-unfollow icon-upload icon-valid icon-video-play icon-views icon-website icon-youtubelogo-imedia-white logo-imedia logo-mediaWhite review-star thumbs_down thumbs_up

Best practices for email campaign testing

Best practices for email campaign testing Jeff Mills

William Shakespeare had a quote for every occasion. This applies to comedies, tragedies, and of course, testing of email marketing campaigns.

Just ask the 37 percent of email marketers who don't test. They will agree that parting from a solid email testing strategy is such sweet sorrow. OK -- maybe it's not so sweet.

In our last eROI study, "Use of Analytics in Email Marketing Campaigns," we demonstrated how important metrics are in making decisions and proving email campaign ROI. One major opportunity to improve metrics and strengthen brand positioning with potential customers is to optimize email campaigns through consistent, careful testing.

In this latest study, we reveal how 623 email marketers are currently using, or not using, testing to improve their email marketing efforts. On the heels of our last study, where we learned nearly one in five are not recording the metrics of their campaigns, it's not too surprising that this study showed more than one-third of marketers are not testing campaigns -- but why aren't they? And for those that are testing, what elements are being tested and what can we learn from them? With 73 percent of marketers planning to increase email as a priority in their future marketing plans, these are the important answers we set out to discover.

To test, or not to test?
A 2006 MarketingSherpa article by Anne Holland proved, through survey results, that testing increases ROI. The responses show that "in every case more than 50 percent of marketers improved ROI (even if only moderately) by testing."

The results of eROI's email testing survey show that 37 percent of email marketers do not test their email campaigns. With what we know from the MarketingSherpa survey, this is essentially saying 37 percent of email marketers don't care to improve the impact and, ultimately, revenue, of their email campaigns. We know that not a single good marketer would purposefully think like this, so we uncovered some of the main reasons for not testing.

Download the full report for the breakdown on reasons for not testing.

Which test through yonder email campaign breaks?
For those that are testing, they are testing a wide variety of elements including design, frequency, calls to action, and day/time sent. Our survey shows this breakdown for email content testing: 85.2 percent test subject lines, 54.8 percent test calls to action, 50.9 percent test designs, 49.1 percent test copy, 41.7 percent test offers, and 36.8 percent test timing of campaigns.

In the MarketingSherpa survey, Anne Holland uncovered which tests generated the highest ROI. Eighty-five percent of marketers she surveyed said subject line tests had medium to high ROI. Ninety percent said testing offers had medium to high ROI, and 95 percent said testing email design had medium to high ROI.

Since subject lines are generally the easiest element to test, it's no surprise that 85 percent of marketers eROI surveyed are doing so. However, with only 42 percent testing offers and 51 percent testing design, huge opportunities are being missed to give campaigns higher ROI. As shown, testing offers and design will achieve a greater ROI than testing subject lines.

Check out the full report for statistics on the best time of day, frequency, and day of week to send campaigns.

Next page >>

Hath not an email marketing campaign testing strategy?

Segmenting the test. Like the goal of your test, the more specific you can get with your testing audience, the more valuable your test will be. Testing within segments can tell you more because of the consistency of the audience. Testing creative is not the only point -- testing your list is just as valuable. Some subscribers will never respond to an offer; find out who those people are and save yourself some money -- stop mailing to them like everyone else on your list.

Single variable vs. multivariate testing. It's not just what you test but how you test. We know that testing campaign elements leads to higher ROI, but we also know not all elements garner the same improvements. Testing can help marketers understand audience behavior and preferences, and also helps identify and solve specific problems in email campaigns. When testing, there are two ways to do so -- one variable at a time, or multiple variables at once. Among our respondents, 43.6 percent of those who test said they always do so one variable at a time, known as a single variable test -- the most common of which is referred to as an A/B test. Thirteen percent of those that test always do so with multiple variables at once, known as a multivariate test. Meanwhile, 42.9 percent use both types of testing, depending on the campaign.

A/B testing. In email marketing, A/B testing is where baseline control content is compared to test content (subject line change, different offer, time of day, etc.) with the goal of improving email campaign metrics.
A/B testing involves sending a differing instance of an email, including a control, to see which single variable is most effective in increasing a desired metric. The test, in order to be effective, has to reach a big enough audience that there is a reasonable chance of detecting a meaningful difference between the control.

One strategy for A/B testing is to follow the order of an email lifecycle: Test names, then try subject lines, then pre-headers, headlines, graphics, calls to action, then site elements (landing pages, shopping cart experience). The point of this strategy is to test the email lifecycle: deliverability to open to click to conversion.

Multivariate testing. Multivariate testing lets you test several changes at once. Accurate results are attainable without having to increase your total sample size, and the impact of each tested element can be identified.

Another important factor to remember is that the bigger you go with your testing, the more you will learn, and the faster you will learn it. Shakespeare was never conservative, and neither should you be when it comes to your testing strategy if you want to see some real results. Don't be afraid to make some aggressive moves in order to break new ground. You could be remembered and imitated for centuries. Or at least be your marketing department's hero.

The test's the thing. Do your email marketing campaigns pass?

Jeff Mills is vice president of products at eROI.

On Twitter? Follow iMedia Connection at @iMediaTweet.

Jeff Mills brings significant experience in consumer behavior, marketing strategy and product development to eROI (www.eROI.com). He has a proven track record of exceeding client expectations in understanding market opportunities, analyzing data for...

View full biography


to leave comments.

Commenter: Chris Sullivan

2009, September 03

Want even more on email testing?

Get our Email Testing Guide for 32 different tests you can do to imrpove right now! --> http://eroi.me/email-testing-guide