Understand qualitative vs. quantitative research

If you want to be a bad product manager, rely solely on quantitative research. Business is about numbers, after all, and there’s a reason you had to learn statistics in school. If you can’t prove something to a level of statistical significance, it must not be reliable. You would never make a decision about a product that’s used by millions of people by just getting input from a few dozen. What people say is not nearly as important as how many people say it.

If you want to be a good product manager, utilize both quantitative and qualitative research for decision making. Numbers are good, though on their own they can not tell the whole story.

Quantitative research is especially useful in product development as a way of confirming findings through qualitative research. For example, you may conduct customer visits or ethnographic research and uncover some unmet needs. You may want to conduct a survey of a wider group of customers to confirm your findings and get more input on specific aspects, such as how important this need is and the cost to the customer of not being able to currently solve this problem.

Many inexperienced product managers have little if any background in any sort of market research techniques. They may rely on the standard techniques — usually surveys and focus groups — whenever research needs to be conducted. Product managers may not even realize that there are other methods available or may not understand why you would use one method over another.

To generalize, qualitative research is usually better for exploring, understanding, and uncovering, while quantitative research is generally better for confirming and clarifying.

If you are trying to discover unmet needs that consumers can not articulate, there is no way that a survey asking respondents to rank items from 1-10 will offer any insight. However, that method would make sense if you have a good idea about priorities for features and you want to get confirmation from a large sample set.

Similarly, if you want to determine whether a market exists for a new product idea and whether consumers would be willing to pay for the product, a single focus group with 6 participants will not provide the information you need. Instead, that would be a useful approach to take if you are trying to understand underlying issues, preferences, and factors that could be helpful in defining a potential solution.

Product managers do not necessarily need to be experts in conducting qualitative and quantitative research, though some experience is certainly useful. More important, however, is that product managers understand the fundamentals of both types of research, their applications and their limitations.

There are many great resources on qualitative research methods; here are two which are recommended:

  • Observing the User Experience: A Practitioner’s Guide to User Research by Mike Kuniavsky. This is a fantastic guide to a huge variety of different user research techniques, especially useful for web, software, and technology products. It is sufficiently detailed to be used as a learning guide, while written succinctly enough to be useful for quick reference. (In the interest of fairness, I will note that this book is published by a division of Morgan Kaufmann, which is an imprint of my employer, Elsevier. However, I have no specific organizational or financial connection to this book. I would gladly recommend it even if it was the product of another publisher or if I was not employed by Elsevier.)
  • Qualitative Market Research: A Comprehensive Guide by Hy Mariampolski. This provides an excellent overview of different methods along with guidance for those actually conducting the research.

Good product managers will use these and other resources to understand more about the different types of market and user research so that they can utilize the right techniques for the right situation.

Translations available:

28 thoughts on “Understand qualitative vs. quantitative research

  1. Jeff:

    Interesting insights, particularly regarding the application of qualitative research for determining unmet needs (“Heart of the Customer”) vs. using quantitative tools to confirm a set of exisiting assumptions (“Voice of the Customer”).

    The former often has to go beyond venues and tools such as focus groups and may require approaches such as “in the wild” systematic observation of customer behavior and usage. (As you know, the focus group environment can skew consumer reaction….)

    Look forward to hearing more from you on this….

  2. “Observing the User Experience: A Practitioner’s Guide to User Research” is a great book. As a usability professional I have to say that I’m glad you’ve pulled this in. User experience and usability folks spend a *lot* of time doing both types of research. So again, thanks!

  3. Jeff –

    Great post, very informative. I think you are bang-on that most PM’s will assume research is done only 1 or 2 different ways, when in fact data comes from everywhere.

    That being said, it always does need to be properly filtered and vetted to ensure the right decisions are being made. Data is great, but “data” by itself isn’t useful until transformed into requirements.

  4. Jeff,

    As you have said, having a well balanced marketing research skill set that covers both the harder (quantitative) and softer (qualitative) aspects of gaining consumer insights is critical. The links to the other resources are good references.

    Glad I found your site. I think the good cop/bad cop approach to writing about product management is very innovative.

    PMB

  5. Jeff,

    In my experience some of the best ideas and innovations have come from sitting on-site with customers and observing them using my products to do their job. It sometimes comes down to a gut feel that cannot be quantified with numbers. Taking the jump from observation to requirements can be scary, but if it’s right you do it.

    Great post!
    Michael

  6. Michael Ray,

    I’m with Jeff here. I agree many insights come from first hand observation and detailed discussions with customers. But how do you know whether these insights are unique to the set of customers you’ve spoken to personally or representative of the market as a whole? You have to follow up the qualitative with quantitative research.

    I’ve written some about how they can work together on my blog as well.

    http://www.userdriven.org/blog/2008/4/13/qualitative-before-quantitative-research.html

  7. qualitative and quantitative decisions are “words of air”.

    Too big analysis, too few production…!

    If you analyze so much of these kind of things you loose the sence and the goal you started for!

  8. Bruce,

    Good question. You need both qualitative and quantitative data; you can’t ignore either. Different products/markets/situations lend themselves more to one than the other. Customers often tell (or show) you things the numbers do not. But, if you ignore the numbers it’s at your own peril.

    One of the things I love about product management is you have to keep up on both (Q & Q), not to mention a bunch of other things. Keeps me hoppin’.
    Michael

  9. The emphasis on qualitative seems very crucial to being successful in product management. As a healthcare professional, I have been a part of quality improvement efforts and the returns in terms of clinical outcomes and cost savings are quite significant.

  10. Qualitative research is what you do when your market doesn’t exist. Once your market exists quantitative research is what you do.

    If you are doing quantitative research, there are times when you can average your market, and other times when you must deaverage your market.

    In the technology adoption lifecycle, the custom app is a market of one, another one, another one, … eight totally separate markets, but only a single buyer in each of those markets. You still need to gate on the population of that client’s vertical, and the dollars you will find there as well.

    The vertical and early market horizontal (IT) can be averaged. But, late market is where you either average and live with commoditization or move to 1-to-1, and deaverage.

    Qualitative trends tell you where the product can go, and where it shouldn’t. Why enter a market that is contracting and undergoing a lot of mergers? Why enter a market that already has more than five entrants? Where is the space where no one has gone yet? Who will your competitors be? These are not addressed via quantitative research.

  11. more clearly when and how qualitetive and quantitative research can be applied in the field of research.

  12. When you bring features to market that are continuous or sustaining innovations, you can use market research, both qualitative and quantitiative.

    When you bring discontinous technologies to the market, you cannot use market research, because the market does not exist yet, so what is to research. Once you have an early adopter willing to fund their productization of your technology, you can use market research to size the vertical that the early adopter participates in. Once you deliver the early adopter’s productization, your market is the vertical market, so you do market research.

    All of that said, normal segmentation practices are suboptimal in all cases where you are selling in a B2B market.

  13. If all you are doing is research, then it would have to be qualitative or quantitative unless you are conducting your research in a lab. If people are involved, then qualitative and quantiative methods apply.

  14. I agree that both qualitative and quantitative taken together yield the best and most risk free solutions. In case of new product development, generating communication it makes sense to have quali precede and quanti ratify. However do u guys see merit in a quanti dovetailing to a quali…and if so in what circumstances?

  15. With a discontinuous technology, you do qualitative and futures/trends research, because the market does not yet exist.

    When you get your clients for the bowling ally, you can consider them to the be address of the market, so you can do qantitative research on the niche market that that client represents.

    When you go horizontal, the horizontal is the address of the market, aso you can do quantitative research on that horizontal.

    When you go into the late market, you can consider it to be the remaining horizontal. You will have several problems in this market that require you to change your platform to SaaS, which provides you with a lot of quantitative, and you will need to move to 1:1, mass customization–or programming down to the individual–, value dispursement and offer bundling, new interfaces focused on moments, and fitness to meaning–all of these opportunities for quantitative, and qualitative research.

    Quant is forecast is strategy. Qual is vision. So anytime that you need a vision, when the linear will not get you there, you need qualitative. It will happen. The problem with vision opportunities is that they are not recognized early enough. You have to be on the rocks before you see that you need to change your vision. To avoid the rocks, preprogram the vision cycle, which in turn preprograms your qualitative research cycle.

  16. Nominate which research instrument would be most effective for each group
    and explain why?
    I can’t understend what it talking about??
    please help me……….

  17. Jeff,

    Wonderfully said and yeah, thanks for the email that you sent me last month. Certainly helped as a newbie in this Product Management area.

    -Mayank Raj

  18. I had a problem to solve, that had to do with attitude.

    how can I measure attitude in research, will I use qualitative, quantitative or both?
    If both which one first?

Comments are closed.