Some people think that brutal murder may be justified in certain situations. Do you think this applies to the following circumstances? (multiple answers possible)
- Career advancement
- Defense of family members
- Drug-induced rage
- Honor killing
- Bad parenting
- None of these
Would you check one or more of the ones apart from “none of these”?
Personally, I would ask the interviewer what he means with “this”. It seems that with “this”, he is referring to the fact that “some people think that brutal murder may be justified in certain circumstances” — grammatically, that is what “this” means in this case. But maybe, just maybe, he is referring to “brutal murder being justified in certain circumstances.” If he means the first, I would definitely check at least two of the options. If he means the second, of course I would check only “none of these”.
So what if you ask this question to 10,000 people, and 100 of them simply refuse to answer the question, while 2,500 check one or more of the suspicious options? Is the interviewer then allowed to conclude that:
- only 100 people had problems with this question;
- 9,900 people interpreted the question as “Do you personally think that brutal murder is justified in certain circumstances?”; and
- therefore 25% of all people would condone brutal murder.
If you, like me, think that the interviewer did a bad job, you might ask him: “If you really wanted to know whether people condone brutal murder in certain circumstances, why did you not ask them that? Why ask such an ambiguous question?”
The interviewer then answers: “If I would ask people directly whether they condone brutal murder, everybody would say ‘no’. So I could not ask that. I added “none of these” to clarify the question.”
To which I would say: “If you know that nobody would check any of the possibilities if you would ask them directly whether they condone brutal murder, doesn’t that mean that you already know that almost nobody condones brutal murder? And if you think that the question needs clarification, don’t you yourself agree that it is badly formulated? Moreover, why do you think that adding “none of these” as an option is clarifying anything?”
Of course, this interviewer seems to believe that quite a few people actually do condone brutal murder, and are just not willing to admit that. So he feels he needs to trick them into admitting what they actually think (according to him) by asking a question that can be interpreted in multiple ways, whereby people interpret it in one way while he can claim that it was meant another way.
Questionnaire-based sociological research is always a bit flaky, and therefore needs to use questionnaires that are as unambiguous as possible in order to draw any conclusions at all. Questionnaire-based research that uses vague and misleading questions is worthless. A sociologist should not try to play the television lawyer who asks questions in a roundabout way and then, when the accused accidentally seems to admit being guilty, shouts out “A-ha! He fell for my little trick!”
Yet the European Commission has published a report that is based on such a vague and misleading questionnaire. This report is on “Gender-based violence” and draws the conclusion that 25% of European citizens condone “sexual intercourse without consent” (i.e., rape). It posed the following question to the respondents:
“Some people think that sexual intercourse without consent may be justified in certain situations. Do you think this applies to the following circumstances? (multiple answers possible).” I.e., the exact same question as I pose above, with “sexual intercourse without consent” substituted for “brutal murder”.
Since for this question 27% of the respondents checked one or more of the options besides “none of these”, the report concludes that “more than one quarter of respondents (27%) [is] of the opinion that there are situations where sexual intercourse without consent is justified”. Not that “27% of respondents think that there are situations where some people (not necessarily they themselves) believe that sexual intercourse without consent is justified”.
When confronted with the ambiguity of this question, the primary investigator who had formulated it, responded like I wrote above: that this was the only way to get people to answer anything else than “of course rape is never justified”, and that they should be able to understand the actual intent of the question, regardless how convoluted its formulation was, because he added the “clarification” of having an option “none of these.”
Is it likely that this question was interpreted differently by different respondents? Yes, it is very likely. All of the questions that come before it are clearly asking the personal opinion of the respondent, in forms such as “Do you agree that…” This particular question, which comes at the very end, is the only one that is suddenly talking about the opinions of other people. The fact alone that this question was not formulated as “Do you think that sexual intercourse without consent is justified in certain circumstances” while all the other questions follow a formulation like that, clearly indicates that this question should not be interpreted as the respondent being asked to give their own opinion.
The Belgian newspaper “De Morgen” noted that the Dutch translation of the questionnaire differed for the Flemish (the Belgian Dutch speakers) and the Dutch. In the Belgian version, “Do you think…” was translated with “Denkt u…”, which is more or less a literal translation. For the Dutch, it was translated as “Vindt u…”, which means “Are you of the opinion…”, which suggests a bit more that the sentence could mean “do you personally agree…” (still very vague, though). The net result was that significantly more Belgians checked one of the suspicious options than the Dutch did. Considering that there is very little difference in morals between Belgians and Dutch citizens, this demonstrates once more that this question tends to be interpreted by different people in different ways.
So what can be concluded from the questionnaire? The Dutch paper “De Volkskrant” argued that formulating questions in such a roundabout way is a common method in sociology to ask about unpopular opinions. And that while the report can be criticized, in light of the other findings of the report the numbers were not unbelievable. They state: “if you ask in the correct way, many people are shown to have pretty nasty ideas about applying sexual force.”
This conclusion is hogwash. If questions on a questionnaire are dubious, nothing can be concluded from them. The fact that the resulting numbers, when viewed in the light of your own beliefs, seem to support those beliefs, doesn’t change the fact that the numbers are meaningless.
I checked the complete report, and contrary to the statements of “De Volkskrant”, the other numbers are actually not supporting the findings for the last question at all. For instance, the question that asks whether the respondent finds domestic violence (which includes rape of a partner) acceptable, is answered by 96% of the respondents with “unacceptable in any circumstances” (that is, against women — evidently only 94% find it unacceptable against men). Raping of a spouse is a form of domestic violence; this means that if you do not condone domestic violence, you do not condone raping a spouse. How can we rhyme the fact that less than 4% of respondents condones rape of a spouse, with the proposition that 27% of the respondents condones rape in general? Is raping a spouse for 23% of the respondents worse than raping a stranger?
The final statement of “De Volkskrant”, that starts with “if you ask in the correct way…” makes me wonder what they think the word “correct” means. For me, the only “correct” way to ask questions on a questionnaire is making sure that the questions are unambiguous.
In software engineering, we use the term GIGO: “Garbage in, garbage out”. I assume this applies to sociological questionnaires too.