How to create and analyze open-ended survey questions

Surprisingly, surveys can be great tools to get rich, qualitative data from many participants. Generally speaking, we think of surveys as quantitative data and numbers, but with careful planning, surveys can also add to our qualitative insights. 

The first time I used an open-ended question on a survey, I thought, “no one will fill this out.” I mean, how many times have you closed out of a box asking you to “please add any additional comments/feedback here.” Even as a user researcher who feels compelled to respond to every survey or feedback request, there were times I skipped this box.

Unfortunately, I was right. Very few people responded to my open-ended questions. For a while, I determined that they simply don’t work. However, there are ways to frame and use open-ended questions in a way that gets more responses. Once I started using these tactics, I was overwhelmed with feedback and then had to tackle analyzing all the data.

How to write open-ended survey questions

Writing open-ended survey questions can feel effortless. Just put a why or how in front of the questions and send it off. I thought this was the key to getting qualitative data from my surveys. However, the process is more thoughtful than this. 

Instead of considering a survey as a list of questions, think about it as a conversation with participants. This process will lead to a more dynamic and open-ended survey, which will gather more rich data. Here are the steps I employ when writing open-ended survey questions.

Think of the goal

Before writing questions for a survey, it is crucial to understand the desired outcome. There is a time and a place for a user research survey, and we want to ensure we are using them correctly. 

Surveys are wonderful for gauging large-scale behavior, preferences, and prioritization. They very rarely will give you enough deep data to elicit motivations or goals. If your survey aims to understand what and how something is happening, you are picking the correct method. We can use open-ended questions to get right at the root of that goal. 

 For example, let’s say we work at a stationery company and we are looking to understand:

• What type of stationery people pick most often

• How people choose their stationery 

• Which competitors people are using to purchase stationery 

While several methods could work here, such as 1x1 interviews with a follow-up survey, we could also send a survey with a mix of closed and open-ended questions to get a more representative sample

Avoid common traps

There are a few traps we can fall into when writing survey questions. I’ve looked over a survey I’ve sent and wanted to facepalm reading some of the confusing questions I’ve asked participants. By creating questions that are too complex or difficult, we can lower response rates. 

The most common traps we can fall into are: 

Leading questions can cause people to answer in a way that is not consistent with how they feel. Instead of using subjective language, be as neutral as possible when wording your questions. The most common leading question I see is:

• Leading: “How satisfied are you with our stationery?”

• Not leading: “How do you feel about our stationery?” Very unsatisfied -> Very satisfied

Double-barrelled questions ask two questions at the same time and usually include the word “and.” When this is a closed question, it makes it impossible for the user to answer. Even when using an open-ended question, people generally only answer one part of the question. Instead of using an “and” or posing two questions in one, make them separate.

• Double-barrelled: “Which stationery companies do you use and what do you think about them?”

• Not double-barrelled: “What other stationery companies do you use?” “How do you feel about your most recent purchase at another stationery company?”

Future-based behavior questions are the most common mishap I see when it comes to surveys or interviews. People can’t predict the future, and most questions that ask about future behavior aren’t reliable. I could tell you that I will cook six times a week next week, but, guess what, I ended up ordering in two times and going out once, leaving me to cook three times that week.

Instead, we can ask about participants’ past behavior to understand what their habits are.

• Future-based question: “How often will you order birthday cards every year?”

• Past-based question: “Last year, how many times did you order birthday cards?”

Mix it up

Just because we are trying to get rich qualitative data from our survey doesn’t mean every question needs to be open-ended. Response rates are better when you have a mix of closed and open-ended questions. I typically use the 60/40 rule, where 60 percent of my questions are closed and 40 percent are open-ended. 

For example, with the goals of this survey, I would consider questions such as:

1. What types of stationery did you purchase in the past six months? [Multiple choice]

2. 1. Birthday

   2. Get well

   3. Thank you

   4. Father’s Day

   5. Mother’s Day

   6. Graduation

   7. Other (left open)

3. What other stationery stores have you purchased from in the past six months? [Multiple choice]

4. 1. Competitor 1

   2. Competitor 2

   3. Competitor 3

   4. Competitor 4

   5. Other (left open)

5. Think about your last purchase; how did you decide what stationery to buy? [Open-ended]

6. What has been the most frustrating experience when purchasing stationery? [Open-ended]

Do a few dry runs

I used only to do dry runs for my one to one interviews with participants but quickly learned that it is imperative to send your survey to a few colleagues to sense-check it. If possible, consider sending it to some relevant friends and family too!

How to analyze open-ended survey questions

You’ve written your open-ended survey and sent it to the world. Now, get ready for feedback. I think about analyzing open-ended survey data in the same way as qualitative interviews. I go through this five-step process when analyzing this type of data:

1. Skim through the data to get a general sense of the responses you have gathered. While doing this, jot down some themes you notice.

2. Develop tags. I use a mixture of global tags (pain point, motivation, goal, need, task) and project-based tags. I come up with some preliminary project-based tags in step one, which are unique to that particular study. 

3. Code the data with the tags that you created in the previous step. I do this by tagging each qualitative response with one or multiple tags. By using global and project tags, you can represent richer detail.

4. Tally up the most common project-based tags that you found and pick the top three. These three tags will be the significant insights you focus on first. 

5.  Once you know the most common tags, pick out a few supporting quotes directly from the survey as supporting evidence.

Using the stationery example from above:

1. We receive qualitative feedback from the two open-ended questions we asked on the survey. I skim through the feedback and start to jot down the themes that I see from the input:

2. 1. For question three: People look at their calendar to see upcoming events and bulk purchase stationery; people wait for sales to purchase; tend to buy many items from the same shop

   2. For question four: Out of stock items; unclear shipping information; shipping delays; quality of the product is not good

3. I develop tags based on this feedback. 

4. 1. Global tags: pain point, motivation, goal

   2. Project-based tags: bulk-purchase, sale, planning, return customer, lacking quality, shipping issues, out of stock

5. Once I have created the tags, I go back to the raw data and begin the tagging process. For example:

6. 1. I tend to wait for a sale to purchase a bunch of stationery at once because it is cheaper. I look at my calendar and think, what do I have coming up in the next six months, and buy all the birthday or celebration cards I think I will need, plus a few extras, just in case #goal #motivation #planning #bulk-purchase #sale

   2. I had a few birthdays coming up, so I purchased a bunch of birthday cards. I didn’t realize where the cards were coming from and how long the shipping would take. It turns out, it took about four weeks to ship but was then delayed, so half the cards were useless, and I had to go out to another store to purchase the cards in time #pain point #shipping issues #goal

7. After this tagging process, I find the top three common project-based tags and focus first on those. I will then write insights based on these tags. In this case, the most common tags are:

8. 1. Shipping issues. Specifically, not understanding how long it will take to ship 

   2. Sale purchases. Where people wait until they know when a store will have a sale and purchase a bunch of stationery at once

   3. Planning. Indicating that people plan about six months ahead and purchase for all of the events they think they will need stationary for in the next six months

9. To support the insights further, I gather two or three quotes per insight to showcase what participants are saying. 

These quotes bring the insights to life and remind colleagues that our users are real people struggling with real problems

Although it can be a considerable effort to analyze open-ended survey questions, it is worth it. You get a large amount of rich, qualitative data, which is especially helpful if you have sample-size skeptical colleagues. These types of surveys can help your teams make more informed decisions based on a mix of quantitative and qualitative data.