The issue arose in an interesting discussion of survey methodology on the AIR LinkedIn group today. Erin Aselas was crowdsourcing solutions to response options on forced-choice Likert items, the preference of her university's administration. This being the Internet, Erin's question gave way to a broader discussion of whether the forced choice was a good idea or not. The discussion hinged on the question of whether, when asking someone to agree or disagree with a statement, "neutral" was the same as "no opinion."
Aside: Yes, I know my social media credibility just evaporated by mentioning that I use LinkedIn. That's where my colleagues happen to be, and the only thing that makes social media valuable is the people you connect to, so I'll live with it. These are good people.
Three positions emerged in the discussion. Neels Fourie argued that:
If it is at all possible, make use of the "forced choice" method (in many cases it is not possible) because many persons go for the middle response (which is normally also the neutral response). . . . But, to have, say 5 response options like "Fully agree', "Generally agree", "No comment", "Generally disagree", Fully disagree" in that specific order will make your statistical results difficult to interpret.John Leonard came to the opposite conclusion but from the same premise.
Forcing people to make a decision about something they know nothing about not only skews the data but frustrates and angers the respondents possibly polluting the rest of their responses. I often see my own responses skew once I feel I'm being manipulated into answering in an inappropriate way.But Brian Lashley argued against the equivalence of the two, coming down strongly against forced choice:
Neutral is not the same as no answer, not at all. I believe strongly in a 5-point scale with a middle option. It is not the same as no answer. . . . I think if you force people to either agree or disagree and don't give them the option of 'neutral,' that dirties up the data more than letting them choose neutral. It creates measurement error because if people actually don't feel that they have a strong preference, they may choose to skip the question or put an answer that doesn't actually represent their real thoughts.I really do think that these are issues that are best taken up with a particular question in mind rather than with a view to a principled, best-practice answer. Aselas having noted that she's asking Likert items is sufficient for us to do that in this case, but I'm a bit leery of generalizing too far beyond that. Perhaps, though, I might consider three useful rules of thumb.
To me, no-neutral-option is akin to this:
"We have your car in black, very dark gray, very light gray, or white. We do not offer middle shades of gray because that is equivalent to not having the car at all."
The more one can assume distances between options are symmetric with respect to the middle value, the more valuable a neutral category is.
I think Lashley is correct with regard to Likert items. But there are some kinds of questions--the contribution to growth example that I used above--where the middle value is not meaningfully described as a point at which the respondent is neutral between two extremes. In those cases, I think we have to be careful that we are not thinking of the item as a ratio variable when it is really just ordinal. We may not know that the middle value is equidistant between the first and last values. I think that's a safe bet with, for example, this scale:
- Very Satisfied
- Satisfied
- Neutral
- Dissatisfied
- Very Dissatisfied
But consider, as options for the contribution to education example in my previous post, this:
- Contributed everything
- Contributed much but not all
- Contributed somewhat
- Contributed a little
- Contributed nothing
The more one can consider the middle value distinctly meaningful in relation to other responses, the more valuable a neutral category is.
Another key difference is that in the first scale, the neutral category marks a qualitative shift from dissatisfaction to satisfaction. Respondents that are on the inflection point between the two are meaningful. In the second example, though, I don't see such a qualitative shift. Moving one category in either direction doesn't say anything fundamentally different in the way that moving from agreement to disagreement does.
I think the example of the car color is more akin to the first scale, and it works there. But the second scale is something of a different situation, and the example doesn't work as well in that case.
If the distinction between neutral and no opinion is important, so is having a way for respondents to distinguish the two.
Finally, we have to be careful with the question we consider here. The question isn't what we mean by these items but what opinions respondents hold when they select these items. We may think neutral and no opinion are different, but if respondents give a polite "neutral" when they are, in fact, indifferent then we have bad data. If our interpretation of that data hinges on treating neutral responses as having a specific meaning, we need to make sure that respondents mean what we think they mean. That probably means having a "no opinion" option separate from the neutral one or an instruction that they should not answer if they have no opinion; assuming that they will not answer if they have no opinion is probably insufficient.
The feedback you obtain from post surveys allows you to gain valuable insight into customer expectations, and gives you the opportunities to deliver consistently extraordinary customer service and improve customer satisfaction ratings.
ReplyDeleteGood Survey Questions