Research participants: What does good look like?

Finding the relevant and right participants to engage in your qualitative research activities is no small feat. Academy Xi's Eric Lutley posed a bunch of questions to User Research Leader Jess Nichols about how researchers should leverage as many sources as possible to achieve this, being aware of potential biases within each of these sources.

Finding participants and patterns in conversations

Eric: Is there a right number in how many people you need to go out and research?

Jess: That's the million-dollar question. The number of conversations needed to build confidence primarily depends on the methodology you use and the different research segments. Nielsen Norman Group suggests testing with five users in discussions with a usability focus. For exploratory research, like interviews, I aim for eight to 10 participants per segment. This number ensures you can identify patterns, similarities, or differences in your participants' responses and allow you to dive deeper into nuances you may discover during research.

Always set expectations with stakeholders. Identify who you did, and didn't interview as part of your research—indicating that you've engaged with all customers when you've only spoken to customers from a specific industry or the highest spenders can bias your findings to your stakeholders.

E: When you're doing interviews, do you sometimes receive information entirely out of left field, causing you to go in a different direction?

J: Absolutely. You're not researching to check a box, the whole point of conducting research is because you don't know things, right? You have to be open to discovery during research and give yourself enough flexibility to receive information from different audiences and experiences. If you feel there's a discovery of note but not enough time to explore, connect with your stakeholders to discuss future research opportunities. Let them know that there could be a lot of insight gained from further exploration.

Bias in participants and insights

E: If you were to leverage an existing fan base of a product through social media, is there a danger of getting biased interviewees?

J: You'd be surprised, I've found that people that are super-engaged with a product often have strong opinions about it because they're passionate. You could also explore the fan bases of competing products and see the difference in the input.

E: If you were to reach out to a fan of a competitor product and discuss things they love and hate, what would you do with that information

J: It's vital to get conflicting insights; it would be a red flag if you didn't. When it comes to considering how to apply insights to the product, we have to take different perspectives; it's not just one point of view that we take into account.

It comes back to the trade-off. If you want to ensure diversity in participants, you need to spend the budget to either pay for a vendor to recruit or find other audiences. If this isn't possible, you need to be self-aware to know you aren't talking to every person I should talk to, but I'm going to set that expectation in my findings.

E: What is your take on services that find participants for you?

J: It depends on the audience you're after. In a previous role, I looked for a specific user type with a particular set of behaviors. The channels that I had available to me weren't helping me access this audience, so we used a third-party vendor to recruit. It meant that we were paying to get access to each participant, but ultimately that process enabled us to uncover the insights we needed to make product decisions. It's an investment, but if you need it, it can be a great option.

Incentives

E: Is there a rule of thumb around incentives?

J: The amount of incentive you should offer depends on the type of audience you're looking for - easier to recruit participants for quick research will be lower in cost than harder to recruit for participants or longer sessions. If you're finding that you aren't getting the type of participants you need, you could always reassess the incentive you're offering during your research period.

E: What are the best ways to get cheap or free participants?

J: There's a trade-off to consider with cheap or free participants because you won't be able to be picky with who you choose. Less choice may lead to less rigor in your research - so be sure to set expectations with stakeholders with whom you did and didn't include in your research. In saying that, some of the ways you can source cheap or free participants are:

• Friends, family, and personal social networks.

• Leveraging company CRMs or databases.

• Posting on your company's social media platforms.

• Connecting with digital groups in relevant communities.

• Paying for targeted ads on social media platforms.

E: How do you prevent the disconnect between the answers you receive and the users' final actions?

J: It's essential to set expectations around attitudinal insights versus behavioral insights. I love this great quote from Margaret Mead:

As you capture your insights, make sure you're considering what is behavioral (i.e., what they do) and attitudinal (i.e., what they say). It's also important to be careful about what people say they do. Many people make the wrong assumption that this is behavioral when it's attitudinal because it's the participant's perspective and bias on what they do. As a researcher, you aren't able to tell the future; you can only determine what is most likely to happen based on what you learn through research. If your hypothesis ends up being incorrect, take the lessons learned from the study and outcome and incorporate that into your next project.

Finding the right participants is integral for researchers to ensure they can accurately answer the open questions they are investigating within their research. Tapping into all possible networks ensures you find the best participants for your research needs. It's essential to set expectations with your stakeholders on who you did and didn't engage as part of your research.