Writing survey questions can seem like a walk in the park until you realize how many ways there are to do it poorly. From leading questions to hitting a double-header in just one question, there are a number of things to watch for when putting together your survey questions.
Bias
I’ll touch on this briefly because I have a full post about eliminating bias from survey questions already. If you’re unfamiliar with bias in survey questions, the general principle is that by using any type of positive or negative language in the question, you’re automatically putting the person reading the question in either a positive or negative frame of mind. This, in turn, affects the answer they’d give. It can sound awkward at first to write a truly unbiased set of questions, but if you’re looking for truly unbiased responses, you’ll want to take the time and effort to achieve the unbiased nirvana.
Before we go on, here’s an example of how easily a question can become a biased/leading question.
Double-barreled questions
I think I’ve seen more double-barreled questions than biased questions in the past few years I’ve been working in market research. What’s a double-barreled question? Simply put, if your questions has an “and” or an “or” in it, it’s likely double-barreled. Here are some examples.
Please rate your experience with communications and follow-up from customer support. You may have had a poor experience with the communications from the company, but a good experience with the follow-up from customer support. It’s entirely possible that the original writer meant for the respondent to rate all communications from the customer support team, but, if so, they should have left out the “follow-up.”
How likely are you to recommend the product to a friend or colleague? This is more commonly written as “friend or family member” which makes more sense than “friend or colleague.” I’m not a fan of the former, either, because there are some things I might recommend to friends but not family members, and other things I’d recommend to family but not friends. However, worse is friend of colleague, because we typically have different relationships with friends versus work or school colleagues.
Is your current living space too noisy or too warm for you? What’s a respondent to do if their living space is quiet but feels like a hot yoga class? Or noisy but cold? (And yes, the question also uses biased language.)
The easiest solution to double-barreled questions is simply to split the question into two. This clarifies matters for the respondent and for analysis later on.
Poorly worded or marked scales
Have you ever seen a question that made sense, but the scale that went with it didn’t? Here’s an example.
Please rate your satisfaction with the product.
Agree — Disagree
Oops! Looks like the scale needs to change to a satisfaction scale instead of an agreement scale!
Another issue with scales that can happen is:
Please rate your satisfaction on a scale of 1 to 5.
Is “1” best or worst?
It’s actually easy to leave scales unlabeled: you’re so familiar with what you want, that you forget to add anchors or words to the scale. To combat this error, be sure to have someone else review your survey before it goes out to your audience. Just like writing anything else, fresh eyes can catch errors you’ve had for awhile but didn’t see because you’ve spent so many hours writing and programming the survey.
Robotic language
It’s so easy for us to write business-speak. Or maybe that’s just me. But I’ve seen so many surveys that use extremely formal language that made me feel like a machine was talking to me instead of a human.
I will admit, it can feel like the I Love Lucy episode where Lucy is taking ballet class and is told back straight, chest out, butt tucked, arms bent but not too much…when reading through do’s and don’t’s of question writing. I promise it’s possible to have natural language questions without using biased language. For example, you can either say, “Please rate the effectiveness of the product,” or you can say, “How effective was the product?” You can say, “Please rate your level of satisfaction with the solution you were provided by the customer service department,” or you could say, “How satisfied were you with the customer service department’s recommendation?”
I personally tend to get over-wary of being biased in language in questions, so it can feel like opening with “how satisfied” or “how effective” is putting a positive spin, except you aren’t saying, “How incredibly” satisfied or effective.
Not giving your audience an out
Now to address my personal pet peeve: the lack of either a “don’t know” or “not applicable” option. This is a pet peeve of mine simply because there are cases where you really, really want to force someone to provide an answer, so you don’t want to give them an out, but then again, as a survey respondent myself, I’ve been honestly frustrated when I didn’t have an answer and was forced to provide a response. Sure, it may seem like your audience should know the answers to the questions being asked, but first, a good screener can help make sure the people answering your survey are already knowledgeable enough, and second, even if they are knowledgeable about the topic, maybe they aren’t in the know about that particular item.
For example, let’s say you’re trying to survey a bunch of pharmacists about a particular drug. You’ve already screened in that they know about the drug in question and have filled the prescription before, but they might not know about all of the questions from patients, because they work with another pharmacist (or two or three) and don’t get to speak to every single customer when filling the prescription.
Always include a “don’t know” or “not applicable” for your questions. One concern many have had (including myself) is how many end up having to select that in a survey where every response matters. However, if your screener is good, the number who end up selecting that should be pretty low (say about 1-2% of your respondents).
Too wordy
I love writing. I love words. But this means that sometimes, my questions can feel too wordy. Introductions to new survey sections can be too long. Dare I say it, even some of my blog posts might be a smidge on the long side. When it comes to survey questions, keep it as simple and as short as possible. Don’t lose your audience by explaining too much, or providing answers that are too long to read.
What questions do you have about survey writing? I’d love to help! Just leave me a comment or tweet your question to me (@zontziry), and I’ll address it in a future survey writing tips blog post!