Research repositories in 10 minutes

At both UX Australia’s Design Research 2020 and Rosenfeld Media’s Advancing Research 2020, Dovetail’s CEO, Benjamin, presented a talk called ‘Research Repositories in 10 minutes’. He spoke about what a research repository is, what problems they are trying to solve and covered some principles to guide decision-making if people were planning to set one up for their own organization. Below is a lightly edited transcript of the talk with an additional Q&A at the end.

What is a research repository?

It started in 2017 with the article Democratizing UX published by Tomer Sharon, then Head of User Experience at WeWork. In this article, which went on to become extremely popular, he coined the terms atomic research and nuggets. A couple of years later, Matt from Microsoft published an article called ’How Microsoft’s human insights library creates a living body of knowledge’. This was released in June 2019 after Microsoft built its internal system, HITS. In the article, he talks about collaboration with researchers and how insights should be created throughout the project rather than a big cliff to climb up at the end of the project.

The period of June, July, and August of 2019, was a very hot time for research repository stories. Sarah from GitLab wrote ‘Why we built a UX Research Insights repository’ and then Etienne from Uber wrote ‘The Power of Insights: A behind-the-scenes look at the new insights platform at Uber’, Kaleidoscope. Rosenfeld Media also held a video conference on the subject and Etienne shared that Kaleidoscope was built in people’s spare time over months. So these are the internal tools that organizations have invested in. After hearing about these platforms and also talking to our customers, we landed on the following definition of what a research repository really means. We think it’s a centralized, searchable database of research and insights that the entire company leverages to make better decisions. The whole point is to allow everybody to uncover user research insights and explore the context that led to them.

Why set up a research repository?

The question is why does everyone seem to want to set up a research repository and what exactly are the problems they are trying to solve? After talking to our customers and reading through the articles, I’ve summarized some of the problems to share with you.

Problems repositories try to solve

Teams are siloed

Research is conducted differently across teams and departments, especially in large organizations. Different methodologies, tools, and formats for insights can cause silos. Here’s a quote from a customer interview we conducted last week:

Research is repeated

New team members join and ask the same questions over and over. Similar projects are run repeatedly. Sherif, a Product Manager at Atlassian said:

A research repository should allow an organization to leverage its pre-existing customer knowledge as it builds over time.

Reliance on tribal knowledge

Undocumented knowledge means you have to know who to talk to. When that person leaves the organization, they take their knowledge with them. We refer to this as tribal knowledge, and relying on it in any organization is dangerous. Another quote from a customer:

Reports aren’t standardized

Because everyone works differently and we’re in a gestation period for the discipline of user research, research reports aren’t standardized. We’ve seen Powerpoints, blog posts, Confluence pages, and more. Without a consistent format, it can be difficult to leverage the insights from the past. This is what Tomer Sharon was talking about—having consistency for the formats makes it easier to store, search, and leverage that past research. Reports should also be more compelling, interactive, and include different media. All the context for the research should be linked in the report itself.

Data is spread across tools

Lastly, research data is spread across a variety of tools. Most people are familiar with Google Drive, Dropbox, Airtable, Word documents, and of course, sticky notes. While these tools are all great in their own right, they are limited in that you can’t really search them. When they pull that wall of sticky notes down, they are gone. Sticky notes also require people to actually be in the room, which is not so good for remote work. Another customer indicated:

These appear to be the most common problems that motivate people to implement a research repository. We also found an article from Jakob Nielsen of Nielsen Norman Group from 2005, indicating that when people have to locate existing usability reports, they’ll fail, or they won’t know what to look for because there is no single place that lists usability results. Worse, if past project owners are the only source, you risk losing this when they leave the company or their roles are reassigned. While in this example, Jakob is specifically talking about usability reports, he predicted the whole trend 15 years ago, talking about the lack of a single place and the institutional memory when people leave the company or move projects.

5 principles to guide your decisions

Let’s say you’re sold and you want to set up a repository, and you feel your organization will benefit from it. Here are a few principles to keep in mind to guide the process.

Principle 1: Retrievable

We all know that researchers are busy and often outnumbered. The whole purpose of a repository is to help scale a research team to service the entire organization. This means that the repository has to support the ability to retrieve answers to questions in a self-service way. As a tangible example of that, something like search should be effortless, free of complicated queries, filters, or views.

Principle 2: Approachable

We also know some stakeholders don’t feel comfortable interacting with research, especially if there is clunky software that requires training. We should also encourage everyone to make data-backed decisions. The head of research for one of our customers shared their Dovetail project with the board of their company, like a big public company, and had the board kind of playing around with it. She recounted that it was a huge win because the members of the board felt connected to their end-users.

Principle 3: Traceable

Evidence helps to build trust in the research team. I’m sure you are familiar with bringing in a stakeholder—for example, a developer—into a usability testing session or interview and seeing their first-hand reaction to observing a user. Traceability between the insight and where it came from is very important and helps to keep biases in check by providing that extra context for people to explore the data on their own.

Principle 4: Accessible

We know that researchers want to build participation to create empathy with end-users. It’s a very common thing that researchers like doing, but it’s counterproductive if you have the research data locked away somewhere. The idea of the repository is to create a level playing field regarding access to data. Tangible examples are features like open by default permissions and one-click sign-on for easy authentication.

Principle 5: Secure

Lastly, we live in the age of regulation like GDPR and the California Consumer Privacy Act. Your repository will store sensitive data, so it’s important to consider storage policies, deletion policies, features like encryption and anonymization, and further requirements this space.

A few other things to think about

There are a few other things to think about which might prompt you to ask further questions. Things like, how will you get buy-in from the org? The repository won’t be successful only one or two people using it. Who is going to have access? If you are going to allow everybody to have access, will they create insights or just be viewers? How will you migrate past research data? Will you pick a time and say that from here on out we’ll be using the repository? What kind of data will you store? So, for instance—interview transcripts, video recordings, usability testing notes, maybe survey responses, and NPS? And who will manage the taxonomy? And guard it and make sure it doesn’t turn into the Wild West? Who will standardize and be the guardian of the data?

Questions from Design Research 2020

Benson Low, UX Research Manager at REA Group asked :

In response, Benjamin indicated that the folks at Dovetail have been inundated with people talking about finding ways to set up their repository, looking for tools to help them out, etc. Most organizations can’t afford to do what the big guys have done and build their own internal platform. They appeared to be either big teams working on the project or longer-term part-time projects that the researchers initiated and received developer assistance to help them actually build-out. Organizations trying to solve their research problems with tools makes a lot of sense. It seems to be a reaction to the general rise of research inside companies. As a designer, Benjamin highlighted the parallel to what happened to design when designers managed to get a seat at the table.

Another question that was asked was:

There are so many different methodologies and formats of data including video, text, transcripts, notes, survey responses, both structured and unstructured. Importing it and transforming it into a format that makes sense for whatever tool you’re deciding to use is difficult. We see a lot of customers just draw a line in the sand and say we’re going to start from here. Or maybe they do a gradual rollout and use it for interviews but not for survey responses. When importing data, often unstructured qualitative data, you end up realizing just how much you actually have. The question is whether it’s really worth it, which is up to each organization to decide how valuable that stuff is.

Questions from Advancing Research 2020

A lot of our customers face the challenge of balancing the richness of having demographic information available versus anonymization. Obviously, if you can put a face to the name, then that helps to build empathy with stakeholders and makes it a lot more real for them. So, I would always try and encourage organizations to find a way to do that within the constraints of privacy and regulation. Whether that’s getting participants to sign waiver forms, or combatting it by choosing the right software that will allow you to achieve that. When customers can’t get approval, something we’ve seen is customers completely or partially anonymize the data. They might look at mapping the data from another tool that holds a code or a key for the data they put into Dovetail.

This is heavily dependent on your organization. We’ve found that some companies are fine with storing personal information, because they have various checkpoints and a secure vendor assessment process. But other companies are stricter, and this often depends on the industry.

My personal opinion is that being able to put a face to the name and getting your stakeholders to resonate with real people is valuable. It’s just a matter of how can you make that work within the constraints of your organization. Fundamentally research is about the people, it’s about the humans behind the software, and I think if you start to take that away, then it loses a lot of its impact.

Typically you’ll find success when the organization is bought into it. If you think about the whole point behind a repository, it’s to allow the research organization or research function to scale. In that regard, it’s about allowing the insights to go further than they would if it was just a researcher that had to push them out.

I do believe in the whole idea of a self-service platform – some of our customers even refer to Dovetail as a ‘portal’. People need to be able to just log in and find answers to questions they have, that will ultimately help them make better decisions. We’ve seen this with quantitative tooling like Google Analytics and Mixpanel, where they can roll out access to analytics to the whole organization. Product managers and designers use access to these tools to make decisions, and I think it’s the same sort of thing with qualitative data.

As a small team inside a larger organization, having buy-in from everyone is important, and having stakeholders leverage the data that’s there. If you don’t have that then you are facing the same issue of silos or research existing only within the product team.

We’ve found that a lot of discussion around research repositories is specifically talking about insight repositories, where it’s like a CMS that you put your PDFs in at the end of the project. We don’t feel like that’s a great evolution over current file storage software or existing CMS tools.

We strongly believe in the traceability of starting with raw data—whether that’s an interview transcript, survey responses, or whatever methodology you’re doing—and then building these insights out over time, rather than it being this singular point at the end. Some of our customers have started using Dovetail for analysis and then they’ll put screenshots from Dovetail into their PDFs. At some point, they realize that it doesn’t make a lot of sense. They may as well just stay in the tool.

PDFs are popular because you can email them to people, but they’re not interactive at all. You can’t put videos in there and playback videos of usability testing sessions. I think it’s about trying to understand where the weaknesses of your current form of communication exist and maybe pivoting away to embrace the richness of a web product. It’s also important to identify how you can benefit from a research repository when you’ve got the raw data there as well as the output. I think that’s valuable and often overlooked because there’s plenty of CMS systems around like WordPress that would make a great research repository, if all you cared about was insights.

The value is being able to go from the place holding all the data (video recordings, demographic information, survey responses) and then connecting it to insights which allow you to go back into that data and build a deeper connection.