ellipsis flag icon-blogicon-check icon-comments icon-email icon-error icon-facebook icon-follow-comment icon-googleicon-hamburger icon-imedia-blog icon-imediaicon-instagramicon-left-arrow icon-linked-in icon-linked icon-linkedin icon-multi-page-view icon-person icon-print icon-right-arrow icon-save icon-searchicon-share-arrow icon-single-page-view icon-tag icon-twitter icon-unfollow icon-upload icon-valid icon-video-play icon-views icon-website icon-youtubelogo-imedia-white logo-imedia logo-mediaWhite review-star thumbs_down thumbs_up

Qualitative vs. Quantitative Testing

Qualitative vs. Quantitative Testing Reid Carr

The goals for testing advertising creative are typically pretty straightforward. Advertisers measure impressions, impact, click-through and conversion rates over a relatively large sample size. These values are determined statistically and are relatively irrefutable. The questions are to-the-point, and the answers are measurable and specific.

Though it can also yield breakthrough results, qualitative testing, on the other hand, is much murkier. Qualitative testing is done in a less-structured format with fewer participants. Its goal is to identify the underlying sentiment, rationale or motivation of the target with respect to the advertiser’s brand, product, site and/or category.

Advertisers can obtain qualitative feedback from various formats -- one-on-one interviews, usability testing, focus groups, open-ended surveys and general contact forms. Each has its own benefits and challenges, but all should yield positive end results if used appropriately.

Qualitative feedback formats

Typically the first, cheapest and easiest place advertisers and marketers turn to gauge how something looks or works is to solicit feedback from peers, parents and friends. This should be done with caution.

Often when an office mate or friend is asked, “Does this make sense to you?” the guards come up. He or she may be anxious to avoid appearing “stupid,” want to agree with and encourage you, or simply not know why you are asking. Additionally, this person may not be part of the target market anyway. Advertisers and marketers should set realistic expectations on what they are trying to accomplish with cursory feedback received from friends, family and colleagues.

One-on-one interviews are typically very in-depth and are conducted anonymously to gather some of the deepest insights into a specific target’s mind. These interviews can help in pinpointing good media to use to reach the target, as well as identifying specific, targeted messaging and tone to speak to the target’s “pain.”

Usability testing is also done in a one-on-one format; however, it is usually performed in a lab where the subject can have their actions monitored. Tasks are specifically outlined, and feedback can be very direct. Advertisers and marketers use this opportunity primarily to plot a preferred course of action from when the user is working with your creative, but secondarily to obtain insight into branding and messaging.

Focus groups are an intense research method that can yield extremely interesting feedback. This venue -- that brings together a few well-manicured target groups to spend several hours in a room together to discuss a given topic -- provides an opportunity for the target to interact with other like-minded individuals in order to draw out deeper issues about your creative and/or product. Focus groups can be tricky to coordinate and moderate. This is one of many formats of qualitative research that should be left to a professional to elicit unadulterated content and analysis.

A final category of qualitative research traditionally used in creative testing is surveys and feedback form. Surveys can be done either pre- or post-creative to get a sense of the user’s perception of the creative. Here, marketing and advertising can ask specific questions about where the target saw the creative, what he/she thought of it, how he/she felt, et cetera. The information is scored and any open-ended feedback should be reviewed for context (for example, were comments encouraged by preceding questions?) and should also be scrubbed for useful data, then aggregated. The trouble with surveys is that answers may be “led” by previous questions or formats and, therefore, be a little more superficial; caution should be taken in crafting them.

Improving the quality of feedback

In executing any of the research methods above, marketers should make sure they identify the target, establish goals, determine sample size and consider context.

Know your goals: First and foremost, when setting out to perform qualitative testing, marketers and advertisers should approach it with the information in mind that they are trying to receive. Many people make the mistake of asking questions out of sheer curiosity rather than start with the end in mind. Marketers that know what they are going to do with the data can be more efficient in the questions asked, and ultimately, in the data they have to sift through. If you don’t need to know it, don’t ask it. It will only breed distrust in the subject, as well as require more work in analysis.

However, it is also important to keep an open mind. In testing the usability of a website, marketers or advertisers may develop a set of heuristic tests they wish the user to take with the ultimate goal of identifying where they get stuck and what should be made clearer. What is often neglected is that in those same usability tests, information can be gathered about messaging and user interests if marketers stay open to it.

Know the target: In any of the above formats, knowing the target is an essential step in improving testing results and making them much more actionable. If you know your target down to a shoe size, odds are qualitative tests will be more accurate. When the right individuals are identified participants will start to echo similar comments and drive home certain issues.

Pick the right sample size: Too broad a sample will dilute and confuse the responses. Too diverse and broad a sample and the answers will also follow suit in those respects. With the right, targeted sample size and consistent and revealing answers, advertisers can obtain better, clearer results and immediately and confidently take action.

Understand the context: As I mentioned earlier, upon receiving feedback, marketers should immediately consider the context from which it came. Many people overreact to feedback they receive on an ad hoc basis -- for example, a form that was submitted through the website that perhaps says that the user had trouble finding something for which he or she was looking. Immediately reacting to that user’s pain runs the risk of solving for the wrong denominator. Not to say that the user’s feedback was not valuable, but the marketer/advertiser does not know how long the person looked for that item (he or she may have gone directly to the form to complain), what the individual's internet skills are (he or she may have just gotten their internet connection) or whether he/she is a target customer (the marketer/advertiser may be targeting a very different type of user).

Context is also something that can be fabricated and/or misinterpreted. In conducting focus groups, for example, the tone of the group can often be altered by the moderator, a loud-mouthed participant, the geographic location from which participants are drawn or even the room in which the group meets. Those factors should be anticipated in advance, as well as carefully considered when reviewing the results.


Smart qualitative research requires up-front planning -- knowing the target, setting goals, determining sample size and considering context. Marketers should look for the big idea when testing qualitatively. Performing qualitative research is an opportunity to get into someone’s head and identify how he or she really feels about the goals advertisers have set forth. The best creative can come directly from a participant’s mouth. Upon revealing target contacts’ pains, trusted sources of information and true feelings about the creative put in front of them, advertisers and marketers can get some very clear directions on what changes to make with which to forge ahead.

Reid Carr is president of Red Door Interactive, helping clients -- such as the San Diego Convention Center, SkinMedica, Benetrac and Sharp Systems of America -- to lay out strategies for their online presence and interaction activities. Before founding Red Door Interactive, Carr formed the interactive arm for the San Diego-based PR agency, McQuerterGroup. Prior to that, he was chief operating officer and accounts director at PBJ Digital, a bi-coastal Interactive development and incubator shop in Los Angeles; before then, Carr handled account management in both the Venice and Playa del Rey offices of TBWA/Chiat/Day. He has a BA from the University of Oregon's advertising program.

As Red Door Interactive's President & CEO, Reid is there for clients and employees alike. Having began his career in advertising, Reid appreciates the integrity of the brand, but focuses on the fact that what we do for clients has to make them...

View full biography


to leave comments.