How to select the right method in UX research

The other day a colleague asked, “what is the difference between qualitative and quantitative user research?” This is typically answered by saying that one gives you numbers; the other gives you more in-depth insight.

It sparked a bit of a debate. Is it that easy, and is that the whole story?

Defining qualitative and quantitative

Let’s start with the formal definition of qualitative and quantitative research.

Quantitative research is formally defined as producing numerical data which can be expressed as stats, percentages, etc. Quantitative research is typically used to answer specific questions, such as “what percentage of our online audience is over 50?”

Qualitative research on the other hand, is concerned with collecting textual (word) data which is used to identify themes and patterns. Qualitative research is typically used to answer broad questions, such as “what features should our online forum provide?” UX methods are categorised as either qualitative or quantitative. Techniques such as user interviews are felt to be firmly qualitative whereas surveys are the opposite and are defined as being quantitative. 

In UX terms the NN Group’s article defines qualitative methods as being those that help answer questions about “why and how to fix”, whereas quantitative methods address questions of “how many and how much.” UX Matters gives a more detailed review of the differences between qualitative and quantitative methods.  

But is it really that clear cut?

We can sometimes use the same method to produce qualitative or quantitative data depending on how it is applied. For example, interviews typically produce qualitative data about opinions, context of use, attitudes and so on. However, if large numbers of people are interviewed using controlled questions (with response options that can be coded), then the resulting data can be statistically analysed, and is therefore quantitative.

There are also other dimensions to consider as well as the qualitative/quantitative axis. For example, the NN Group article assesses UX techniques in terms of whether they are measuring what people do (behavioural data) or what people say (attitudinal data). Again, depending on how they are used, many UX techniques can produce data that is more or less attitudinal or behavioural. For example, interviews can focus on ideas, opinions and attitudes (attitudinal) or can be weighted more towards exploring context of use, tasks and goals (behavioural).

So, how do you decide which method to use?

At Nomensa, we tend to use mixed-method approaches. Each user research technique provides a different perspective and the trick is in combining them and using them in  a way that ensures your research goals are met. For example, we may conduct some in-depth interviews to identify possible barriers to adopting a new online service, and then conduct an online survey to get an idea of the numbers of people affected by those barriers.

There are several steps involved in identifying the right research method or combination of methods.

1. Understand the problem.

Often, what seems to be the problem is misleading. For example, in “The wrong kind of problem” my colleague Emma Chittenden shows how trying to improve online assistance in order to reduce calls to help centres, is often a case of looking at the wrong problem. To truly understand the root cause of a problem, an external perspective is needed that considers the overall business goals and strategy, as well as organisational factors. Understanding the problem should not be underestimated!

2. Ask the right questions. 

This is about defining the research goals correctly. The focus should be on “deeper” questions rather than more superficial aspects. For example, rather than asking whether users like the online forum, it may be more appropriate to question whether an online forum is the best way of meeting the organisations objectives.

3. Identify the data that will answer those questions. 

Are the questions more exploratory (likely to be addressed by qualitative methods) or more detailed (likely to be quantitative)? Do you need to understand what people actually do or what their opinions are?

4. Select the research methods that will provide the necessary data.

In general terms, we have found that methods which produce large amounts of data (quantitative methods such as A/B testing or surveys) help to evolve or optimise something. By their very nature, this type of research has to be quite controlled as otherwise the data is insurmountable.

In comparison, although qualitative methods (such as usability testing) can also be used to evolve and optimise the user experience, these are the techniques that are more likely to produce revolutionary change and provide a completely new perspective. Only through techniques such as interviews or contextual research can you identify the underlying user needs which can lead to a whole new solution.

At Nomensa, we use a set of graphs (similar to the one in the NN Group article) to match research methods to data needs and research questions. Each graph provides detail about how the research method can vary along the dimensions of qualitative / quantitative and attitudinal / behavioural. An example graph is provided below for user surveys.

Figure 1: Example graph showing the types of data that can be collected through user surveys.

These graphs allow us to easily identify the right combination of user research techniques to provide the data needed to answer clients’ questions. The graphs also help define exactly how to use a technique to obtain the data needed – for example, how can a survey be used to obtain more quantitative data?

In summary

Selecting the right research method is absolutely critical if you want to get the data you need to make strategic design decisions. It is not as simple as deciding whether you need “numbers” or “insight”. As we have seen there are a number of dimensions to consider and different ways to use each technique depending on your research goals.

Spending time in understanding the problem and then planning and carefully designing your research approach will pay off in terms of the quality of data and insight that you will get back.

—