Writing unbiased survey questions

Last week, I wrote about the most important step for survey writing: acquiring a razor-sharp focus for the study being designed. At the end of that post, I promised that this week’s survey design tip would focus on some tips for writing survey questions. With that in mind, let’s look at the first set of tips to address what I feel is one of the easiest – and most common – errors for survey questions: biased language.

To bias or not to bias, this is NOT the question

What is bias in a survey question? Also called “leading,” it’s any kind of language that consciously – or unconsciously – skews respondents to either a more positive or a more negative response to the question. This is also extremely easy to introduce into a survey without realizing you’re doing it. Here are examples of common survey questions that would be classified as leading:

• How much did you enjoy your experience? (Positive bias, leading participants to respond more positively.)
• How satisfied are you with the product? (Positive bias, similar to the previous question. Granted, in this case, it’s a little less positively biased than the previous question, but it still assumes a positive stance to the question.)
• A recent poll indicated 74% of respondents were unhappy with the latest government policy. How do you feel about the policy presented? (Negative bias, as well as satisficing bias – respondents are more likely to answer negatively because the question was posed negatively, and because if so many others were feeling that way, perhaps they should, too.)

I have a favorite YouTube video that does a fantastic job showing how surveys can be quite leading – and other biases that can be introduced while taking a survey.

Now, I have been in client meetings where the client did want leading language in the survey for any given number of reasons. However, using biased language in questions results in biased results, which, if results are being used to make business decisions, can easily and quickly lead to the wrong decisions being made.

Wash that bias right out of your survey

Let’s look at the three questions listed above and re-write them to eliminate bias from the questions.

1. Biased: How much did you enjoy your experience? Unbiased: On a scale of 1-5, where 1 is worst and 5 is best, please rate your experience.
2. Biased: How satisfied are you with the product? Unbiased: Please rate your level of satisfaction with the product.
3. Biased: A recent poll indicated 74% of respondents were unhappy with the latest government policy. How do you feel about the policy presented? Unbiased: Please rate your level of agreement with the policy recently presented.

I think one of the reasons questions end up biased is because we’re often told to make our writing more active – in resumes, in blog posts, in anything – because that will make readers more engaged with the text. However, in surveys, you want the writing to be unbiased so that you are more accurately capturing participants’ true sentiment.

Another way to remove bias: the way the answer options are listed. But I’ll address that more next week.

What tips do you have for removing bias from survey questions? I’d love to hear from you – leave a comment below!

Tips to wash that bias right out of your #survey! #surveywriting #mrx http://t.co/DggE6U05oA

— zontziry (@zontziry) September 28, 2015