Choosing the right research method
by Terrance Kirkwood

If you’ve been given the responsibility of improving the User Experience of your product, you will soon discover that depending on your reasons, there are many ways for you to collect user feedback and use that data to drive changes into the product. For example, you could be interested in validating a new idea, evaluating the usability of a new product feature or discovering new opportunities from existing user experiences. In each case there are one or more different research methods suited to help you gather the best feedback from your users. And determining the right research method is essential to collecting the best user feedback. Using the wrong research method could result in wasted time and money and even worse, invalid data that makes the user experience worse, not better.

To ensure you are using the right research method, start by answering three questions that will help you match your research needs to the best research method.

  1. what do want to learn?
  2. what is the best way to collect the data from your users?
    what can you you measure?
  3. how can the product be used for your research?
    will you use the product for you research?

The answers to these question can be represented as a 4 quadrant chart, the User Research Matrix, and used to identify the most appropriate research method(s).

  • The answers to the first question, represents the left an right halves of the chart.
  • The answers to the second question represents the top and bottom halves of the chart.

    The result is four quadrants for each answer combination.

  • The answers to the third question represents different options within each quadrants.

Different research methods can be found in one or more the quadrants based on how well they are suited to answer each question.

What do you want to learn?
In most UX related research, you generally want to learn one one of two things:

  • Is my idea worth building?
    a version of this same question can be is my problem worth fixing?
  • How should i build a product so customers love it?
    a version of this same question can be how should i fix the problem so customers love it?

If you need to know if your idea is work building(fixing) then you need to collect quantitative data. Collecting quantitative data can help you understand things such as:

  • Is there significant interest in something,
  • Is there significant concern with something, or
  • Is there significant occurrence of something to warrant taking action.

If you already have the quantitative data to tell there is significant interest, concern or occurrence then you are likely interested in collecting qualitative data. Qualitative data can help you understand how to build your product so customers love it.

Check out Quantitative vs. Qualitative data If you are unclear about the difference between the two data types.

Some research activities are more suited for collecting qualitative data while other are better for collecting quantitative date. And some research activities, like a usability study, are suitable for collecting both types of data. But before you can choose the right one, you still need to determine how you want to collect the data from your users and how you plan to use the product in your research.

What is the best way to collect the data from the users?
Once you’ve closed on what you want to learn. You need to determine the best way of collecting the data from the users. As with the determining what you want to know, you have two options. You are either going to observe what your (potential) customer do or you are going to listen to what your they have to say. Another way to think about this is you are choosing between measuring your users’ behavior and their attitude.

Because what people say they would do in a given situation and what they would actually do in the same situation, can vary drastically, when it comes to improving user experience of products, as a general rule measuring behavior, followed up with an assessment of their attitude is preferred. However each method of measurement has its advantages and drawbacks.

For instance, measuring a user’s attitude can be good for design because it provides insights into user’s self reported mental models and perceptions about the topic.

And measuring behavior generally requires more overhead and planning than measuring attitude. The overhead and planning comes from the setup of the product and environment required to observe the users’ behavior.

After you determine the best way to collect the data from the users you have to determine how the product can be used for the research.

How can the product be used for your research?
Once decisions have been made about what you want to know and how to collect the data from the users the next step is to determine how the product will be used for the research.

In some cases the decision is easy. You simply don’t have a product to use for the research. For instance you are at the early stages of product development don’t have a product or prototype to test. If you do have a product or a prototype, users can evaluate and use it in a natural environment, a controlled environment or some hybrid version of both.

  • No use of the product
  • Natural use of the product
  • Scripted use of the product
    – or –
  • Combination/Hybrid

Natural use provides the most valid feedback but least amount of control over what feedback you can collect. For instance, minimize interference from the study in order to understand behavior or attitudes as close to reality as possible

However, if you are interested in getting feedback on how a user might deal with a rare or obscure error situation, then testing in a nature environment might never present an opportunity to gather feedback. In that case, a controlled environment might be tier.

A controlled environment, such as a usability study, allows you to set up an artificial environment where you can simulate the user working with your product in the real world. This approach has the advantage of allow you to focus the user on specific tasks and probe the user about their thoughts before, during and after they use the product.

Finally, in some case, you can leverage a hybrid environment, both the natural use and controlled environments, to get feedback from users about the product. An example of a hybrid environment would be a desirability study, in which you allow the users to use the product as they normally would but then follow-up which questions about their presences and reaction to the product.

Choosing the right UX research method requires answering three simple questions:

  1. what do want to learn?
  2. what is the best way to collect the data from your users?
  3. how can the product be used for your research?

Most of times the answers to those questions are easy to answer because of the development phase of your product or the resources available to you. Other times you may have options to consider and need to make careful judgments so you don’t waste precious time and money on a research methods that won’t yield the dat you need to help you improve your product.

But once you have the answers to the questions, you can use the UX Research Matrix to determine the best type of research method to use to collect valuable customer data and begin improving your product with User Research.