United States
Contact Us
Search bar for the website

News and Insights

When is Research Misleading?

December 21, 2019

You will learn:

1. The most common research mistakes.
2. Five steps to guide you around research pitfalls.
3. To interpret research with caution.

I recently read an article about the dubious value of opinion research, citing, for example, the bias created by the moderator in focus groups (“Would you buy this drink?”). The article argues that direct observation is clearly a more accurate gauge of behavior and intent (“The consumer bought the drink.”). Who can challenge that actual behavior observed trumps reported intentions?

Opinion research can certainly be misleading at times. And asking questions to measure behavior is only one of the ways to make it so. I can’t count the number of times I’ve cautioned against considering all research valuable or even accurate. But there are ways to ensure that your findings are sound when undertaking research among your clients, your members, or your markets.

1. Know your Goals

I know you’re thinking, “Of course, we need goals!” but, alas, research can be initiated for nutty reasons. My personal favorite: “Everyone else is doing it.” That everyone else is doing it may make initiating a new study an excellent recommendation, but you still must match your research goals to your business goals. Do you define success by a measurable return on the research investment, or do you just want to touch your most loyal clients? Are you trying to guide or justify a specific marketing expenditure or, more loosely, gauge awareness in a particular market? Knowing what you want to achieve is crucial to obtaining the data you need. Detailing the specific information you want to know, even using hypothetical statements of finding, can help you to make your objectives clear. In this case, the cart (what you wish to carry away from the research) truly comes before the horse.

2. Fully Define Your Target Audience

Do you put stock in those general market studies that “rank” your business better or worse than others? Syndicated studies are great gossip and provide fodder for your website’s homepage (“We’re #1 in reputation for excellence for the third straight year!”), but there is limited value in being considered number one for anything if those who provide the ratings do not purchase or even influence the purchase of your services.

When conducting research, or using research conducted by someone else, you need to ensure that respondents include individuals whose opinions you really need to know. Do you want to know what your top 25 clients think, your clients with the highest potential or your clients who seem to be fading away? Are you looking for guidance from prospects for a specific service, in a specific geographic area, or from a certain type of business? If existing research was conducted among exactly the right group of individuals–excellent! If not, you’ll need to conduct your own research to get what matters to you.

3. Select the Best Methodology

The article I mentioned earlier particularly cautioned against using focus groups to understand behavior. And, focus groups are one of the most misused research methodologies. They are a qualitative research method, statistically invalid, which necessarily makes focus groups ill-suited for drawing conclusions about habits or actions. Whether you conduct one session with 10 individuals or 10 sessions with a total of 100 individuals, they are never conclusive. Focus groups are, however, an excellent way to come up with ideas about proclivities or intent that can later be tested with quantitativesurveys. Focus groups can help you discover undetected problems with an ad campaign, potential challenges of a new service offering, or the usability of a website design. But when you want to understand what is most important among a number of choices, what really drives client loyalty, or how to best position your business in a market—these objectives require a quantitative method that can provide the metrics you need.

4. Ask the Right Questions the Right Way

Another issue raised in the article is how easily respondents can be led to particular responses and how hard it is for them to accurately assess and report their own motivations. I agree! When you develop your discussion guide, in-depth questionnaire or survey instrument, you need to make sure the questions are not leading, that your respondents are not primed to answer in a particular way. (In fact, when conducting focus groups, I often ask participants to write down their initial impressions before discussion even begins.) For telephone or in-person interviews, make sure your interviewers are skilled in the techniques that will bring even subconscious motivations to the surface.

5. Interpret with Caution

How do you know if your findings are truly reliable? Even if you’ve clearly laid out your goals, comprehensively defined your target, picked the best methodology, designed an effective research instrument, and used excellent interviewers, the results can still be misleading if your interpretation of the findings is flawed. Reliable interpretation begins with proper analysis of the data, which requires understanding how the target population was selected and ensuring that your resultant data includes the information needed to fuel your conclusions. Perhaps the most common problems are conducting quantitative analyses with too few responses or having response rate that is too low–both of which beg the question: How do the non-respondents differ from those who are included in the research?

So is research misleading? It certainly can be, but by using these guidelines, you can take the necessary steps to ensure that your research will provide the information you need.