Choosing the right research method
by Terrance Kirkwood

Have you been tasked with improving the User Experience (UX) of a product?
If so, collecting data from your users is vital — and equally vital is asking them the right questions so you get the data you need. For example:

  • Are you trying to get a new idea validated?
  • Are you evaluating the usability of a new feature?
  • Are you looking for new opportunities off the back of products you already have up and running?
  • Or is there some other aspect of the product you need to know about?.

First decide which of these interests you; then choose the research method which will allow you to elicit the data you need. If you don’t, you risk ending up with a pile of useless data which doesn’t take you anywhere or, worse still, with skewed or inaccurate data which leads you in the wrong direction so you end up with a worse UX rather than a better one.

  1. What do you want to learn?
  2. What is the best way to collect the data from your users?
    what can you you measure?
  3. How can the product be used for your research?
    will you use the product for you research?

These questions are represented in a 4-quadrant chart — the User Research Matrix (below), which you can then use to identify the most appropriate research method(s). (Note the User Research Matrix is adopted from the 3-dimensional framework presented by Christian Rohrer.)

The first question represents the left and right halves of the chart. The second question represents the top and bottom halves of the chart. The result is four quadrants for each answer combination. The third question represent different options within each quadrant. Different research methods live in one or more of the quadrants based on how well they support answers to each question.

What do you want to learn?
Most UX-related research is directed at finding the answer to one of two questions:

  • Is my idea worth building?
    (aka: “Is my problem worth fixing?”?
  • and

  • How should i build a product so that customers love it?
    (aka: “How should I fix the problem so that customers love it?”

To answer the first — is it worth building/fixing? — then you will likely need to collect quantitative data to help you understand aspects such as:

  • Is there a significant interest in something?
  • Is there a significant concern with something?
  • Is there a significant occurrence of something to warrant taking action?

If you already have the quantitative data to answer one of those three questions, then you are likely interested in collecting qualitative data, as this will help you understand how to build your product in such a way that customers will love it. (Check out my post on Quantitative vs. Qualitative Data if want to know more about the differences between the two data types.)

Some research methods are better suited to collecting qualitative data, others are better for quantitative data,and a third group — like a usability study — can be used for both.

Now you’ve decided which type of data you are looking for, you need to answer two more questions before you can choose your research method:

  • What is the best way to collect the data from the users?
  • How do you plan to use the product in your research?

What is the best way to collect the data from the users?
You have two basic options here, based on whether you want to measure your users’ behavior or whether you want to measure their attitude. If the first — behavior — then you need to observe what your (potential) customer does. If the second — attitude — then you need to listen to what they have to say. Ideally, you would use both methods but this is not always possible in real life.

Measuring attitudes generally gives you useful design feedback as you are tapping into users’ mental models and perceptions of the product as it is, or the ideal product they would like to see. However, your participants’ own accounts of what they like and want can include biases that result in inaccurate data. (Check out my ebook on Customer Interview Best Practices, if want tips on how to formulate good discussion guide questions to the best possible feedback from your interviews.)

Measuring your users’ behavior can give valuable all-round information, but be aware of a potential trap: there can be a significant difference between what people say they would do and what they actually do, i.e., between self-predicted behavior and real behavior. Getting scientifically meaningful measurements of user behavior can therefore be logistically complex and costly as it generally involves creating a research environment in which you can observe how users really interact with the product.
After you determine the best way to collect the data from users, you then have to answer the final question before choosing your research method.

How can the product be used for your research?
In some cases, the decision is easy — you simply don’t have a product to use for the research. This might be the case if you are at the early stages of product development and don’t have a product or prototype to test. If, however, you do have a product or prototype, then you have the option of letting users work with it in a natural environment, a controlled environment, or some combination of both.

The choices are:

  • Natural use of the product
  • Scripted use of the product
    – or –
  • Combination/Hybrid

Each has its distinct advantage.

In the natural use scenario, you just sit back and observe, without interference or comment, how users interact with your product. This gives you very good insight into both behaviors and attitudes in an environment which replicates a real-life user context as far as possible. The downside is that you have very little control over what feedback you will be collecting at the end of the study as you haven’t asked your users any specific questions.

So, if specific feedback on, say, how a user might deal with a rare or obscure error situation, is what you are looking for, the natural use scenario is not right. In this case, a controlled use scenario, such as a usability study, would be better. Choosing a controlled use scenario allows you to set up an artificial environment where you can simulate the user working with your product in the real world. You can then direct the user to focus on specific tasks and give their feedback on these before, while, and after they use the product.

The last of the options is the combination scenario, which brings together elements of both natural use and controlled scenarios. One example of the combination scenario is a desirability study, in which you allow participants to use the product as they normally would but then follow up with questions about their preferences and reactions.

So, choosing the right UX research method requires answering three simple questions:

  1. What do want to learn?
  2. What is the best way to collect the data from your users?
  3. How can the product be used for your research?

Usually, these questions are easy to answer because they are dictated by the development phase of your product or the resources available to you. Other times, you have options to consider and need to make careful judgments that won’t result in wasted time or money. But remember that once you have the answers to these basic questions, using the 3-dimensional framework presented by Christian Rohrer is the best way to choose the right research method for collecting valuable customer data and improving the UX of your product.