Customer Surveys: Open-Ended vs. Multiple-Choice Questions

on September 9, 2019
Customer Experience

As a business owner, you simply can’t overestimate the importance of what your customers think, feel, and say about your product or service. Most likely, you already know the key to discovering this invaluable information: A customer survey.

Customer surveys allow you to reach out to your customer base, probe them for their thoughts and opinions (in a friendly way, of course ), and filter their feedback for common trends and themes. The resulting insight can help you improve your product, make better business decisions, and convert more of your customers into vocal, enthusiastic advocates for your brand. Sounds good to us!

The trouble is, how do you decide on what kind of survey to use? With a seemingly endless variety of formats and question types, it’s difficult to know where to begin for the best results.

In this article, we’re eliminating some of the guesswork for you by giving you foundational advice for creating an effective customer survey, as well as a good old-fashioned compare-and-contrast between two of the most common survey question: open-ended vs. multiple-choice.

Your Starting Point: Net Promoter Score (NPS)

Before deciding how you want to write your customer survey, you need to decide what sort of information you want to collect. If you’re focusing on customer satisfaction, there are several metrics that you can gather. These include CSAT (customer satisfaction score), CES (customer effort score), and finally, NPS (net promoter score).

While CSAT focuses on measuring general rates of customer satisfaction, CES seeks to gather information about how challenging it is for customers to interact with your product. Both of these metrics can be valuable in their own right, but what you’ll really want to focus on is NPS.

Why?

NPS effectively captures a spectrum of customer satisfaction-related insights by asking your customers how likely they are to recommend you to a friend, relative, or coworker. In doing so, they let you know how happy they are and how usable your product is. Maybe most importantly, you also gain insight into your success in encouraging word-of-mouth marketing.

NPS is generally collected by asking, “How likely are you to recommend Product X to a friend, relative, or coworker?”

Survey-takers then have the opportunity to respond on a scale of 0-10. Customers who give you a 9 or 10 are promoters; customers who give you a 7 or 8 are passives, or neutral, and customers who give you a 6 or below are detractors.

In a perfect world, all your customers would be promoters, eagerly sharing the good news of your product with others. But that’s not usually the case. In all likelihood, you’ll find more of a combination of customer types. Your goal is to discover why promoters love you, and why detractors are less than satisfied. And then, you’ll want to turn that feedback into actionable advice.

The Follow-Up Question: Multiple-Choice or Open-Ended?

Your follow-up question is the key to turning insight into action.

If someone gives you an enthusiastic “10,” you can take a guess about why they love your product…but in reality, you need to hear from your audience to get an authentic answer. On the flip side, if a survey-taker gives you a “4,” you’ll most likely be desperate to understand the logic behind their less-than-thrilled answer.

In any case, you need to give them an opportunity to explain.

You can do so in one of two ways: a multiple-choice question, giving survey-takers a pre-set range of options to choose; or an open-ended question, providing them the chance to use their own words to explain their rating choice.

Multiple-Choice: Pre-set Answers for Easy Analysis

A multiple-choice question for collecting NPS feedback for a meal delivery service might read as follows:

What can we improve on?

a) Customer service
b) Quality of ingredients
c) Package design
d) Recipe instructions

The core advantage of asking a multiple-choice question is that you generate a highly controllable outcome: You receive specific information with very little need for analysis. It’s fast and easy, and can give you some actionable insight.

The disadvantage, however, of asking a multiple-choice question is that you don’t really learn why your customers rated you the way that they did. For example, by asking your customer base the question above, you learn that a number of them don’t like your package design, and very few have complaints about customer service. But did you really learn about their true concerns? You don’t understand your customer’s true sentiment.

Let’s imagine that your detractors’ primary complaints are a slow delivery time or unclear messaging. You haven’t given them an opportunity to share that information. Instead, you’ve made assumptions about the logic behind their rating.

Not only that, but you haven’t created a space to allow your customer base to make suggestions or recommendations. Now you’re missing out on a relationship-building opportunity that shows you value their opinion.

Open-Ended: Detailed, Honest Feedback

On the other hand, an open-ended question for following up on an NPS rating question might simply read:

Please explain your answer.

Pretty simple, right?

Giving your customers free reign to give you feedback produces some powerful insight. You may discover common threads and trends in your feedback analysis that surprises you, and help you to effectively turn more of your passives into promoters–and your promoters into more active advocates for your brand.

For example, you may learn that many of your customers are unhappy with a particular ingredient that’s present in many of your meal delivery boxes. This ingredient could be killing off your retention rate, and turning your satisfied customers sour.

But without an open-ended question, you would have remained blindly ignorant of your meal delivery faux pas and left to wonder about dropping subscription rates.

Not only that, but an open-ended question gives your customers space to voice their complaints, and shows that you want to listen.

This can help you to develop effective strategies for improving customer loyalty and advocacy.

The primary challenge of using an open-ended question is, of course, analysis.

Open-ended feedback leaves you with a vast amount of information to read, organize, filter, and most importantly, understand. The larger your customer base, the more amplified your challenge. Take into consideration the fact that many of your customers may misspell certain words and use different phrasing for similar sentiments, and you have hours of data sorting on your hands.

Don’t Sweat It: There’s a Solution

Open-ended feedback produces the not-insignificant challenge of analysis. But relying on a multiple-choice question to gain authentic insight into your customer base leaves a lot to be desired…and may ultimately result in wasted time, money, and resources.

The solution looks to augmented intelligence to help you effectively categorize, filter, and analyze open-ended feedback. Caplena helps you deliver NPS surveys and analyze the feedback, getting you detailed, accurate insight without the massive challenge of trying to perform the analysis yourself. Instead, artificial intelligence does the heavy lifting for you, by using NLP (natural language processing) to evaluate language. At the same time, we help create the codebooks ourselves, using human creativity and insight to establish an effective system of categorization.

Related blog posts

5 Ways to Effectively Measure and Utilize NPS

Your customers have an enormous amount of leverage when it comes to your brand’s reputation. What they tell other people can expand your business quickly or destroy new growth. That’s why it’s so important to measure NPS (Net Promoter Score).

Customer Experience

A 2-Question Customer Survey: Yes, It Really Works

Where do you start with creating a customer survey? While the options can seem overwhelming, keeping your core motives in mind is key to success. Thankfully, there is a solution: A 2-question customer survey.

Customer Experience

Getting to Fully Automated Tracker Evaluation

Trackers provide a strategy for collecting valuable input and data over time for your business. Discover what you can gain by tracking the evolution of KPIs and open-ended feedback on your business.

Customer Experience

5 Tips to Take Full Advantage of Your Text Analysis SaaS Trial

Are you ready to choose your text analysis software? Read to learn how to take full advantage of your trial and set yourself up for success.

Customer Experience

Chart of the Month: Driver Chart

Driver Scatter Analysis is a method for measuring a code’s influence on customer satisfaction (NPS/CSAT). On the Driver Chart, you can see immediately which issues “count” and where to focus your efforts. 

Customer Experience

Datasets for Getting Started in Text Analytics

Would you like to get started with Caplena and try out some datasets?
If you’re interested in some demo surveys with open-ended questions, we’ve got you covered: Here are 5 datasets with free-text feedback.

Customer Experience

Chart of the Month: The Treemap

The whole idea behind Treemaps is very well suited to text analytics data, which is why treemaps are a popular type of chart within Caplena. Learn the origins of treemaps, how to read them, and best practices applied to feedback analysis.

Customer Experience

How to Ask for Customer Feedback – And Get an Honest Response

Customer feedback is about more than knowing whether your customers give your product three stars or five stars (although that is a valuable insight). It’s also about gathering the information that can help you make critical decisions and improvements….

Customer Experience

10 Useful Customer Feedback Categories to Help Give You Valuable New Insight

Most business owners are well aware that open-ended feedback provides invaluable insight into helping them improve their product or service, boost user satisfaction, and even expand their customer base.

Customer Experience

Using Customer Feedback Tools to Close the Loop

Customer feedback is an invaluable resource to your business. By gathering and assessing the sentiment of your customers with an NPS survey, you’ll learn what they’re actually thinking…

Customer Experience

Coop Case Study: Getting Actionable Insight From Mass Feedback Analysis

What do customers think about organic food? What do they like about the newly built store? Why do some customers feel 😊, while others feel 😩? These are questions that Coop, the Swiss Retailer wanted to find out!

Customer Experience

3 Easy Steps to Analyze Survey Responses using Caplena

Today, we will take you through the three steps required to evaluate your open texts survey responses with the help of Caplena, the market-leading text analysis tool!

Customer Experience

The Top 5 Customer Feedback Tools in 2022

Don’t spend hours researching – we’ve done the work for you. Explore the top five customer feedback tools of 2022, whether you’re a consultant, insight team member, or market researcher.

Customer Experience