5 mistakes to avoid at all costs when creating questionnaires

Many students conduct an empirical survey as part of writing their bachelor’s or master’s theses at university and this is now often in the form of an online survey, which unfortunately is also often self-selected via Facebook, Twitter, XING, and other social networks. Since I have a lot to do with surveys myself in the context of research and teaching, I always enjoy looking at such student questionnaires. Most of them show that heart and soul have gone into the questions, which is why it is even more regrettable when the validity of the survey is unnecessarily limited in the end by purely “technical”, easily avoidable mistakes in the formulation of the questions. 5 typical mistakes that I have noticed particularly often in recent years will be shown and discussed below using examples from a fictitious survey.

1. Double question

What is the highest educational qualification you have or are currently pursuing?

This question combines two questions, namely the one about the highest educational qualification achieved and the one about the highest educational qualification aspired to. A respondent who, for example, has the Mittlere Reife (intermediate school leaving certificate) but is currently also studying for the Abitur at night school will have just as much difficulty answering this question as a respondent with the Abitur who is currently studying. If only one answer is permitted, the respondents have to choose one of the two sub-questions, and since it is impossible to know for sure which respondent answered which sub-question, the data collected is virtually worthless. Out of 100 respondents, perhaps 43 will end up stating their highest current degree and 57 their highest intended degree and all wildly mixed up. It goes without saying that one cannot do much with such data.

2. overlapping answer categories

What age group do you belong to?

a) 15–20 years
b) 20–25 years
c) 25–30 years
d) 30–35 years

As in the case of double questions, some respondents have to decide almost arbitrarily where to put their cross, even if the answer categories overlap: As a 25-year-old, does one check answer option b) or rather answer option c)?

Solution: Specified answer categories must never overlap especially if only one answer is permitted. Fortunately, this can usually be ruled out quite easily, however — all you have to do is read through all the questions and answer options again thoroughly (or have them read through) before releasing the questionnaire, taking this rule into account.

3. Far too many questions

If you don’t have an expensive incentive on offer or you don’t survey a particularly motivated group of respondents, you shouldn’t assume that respondents are willing to invest more than 5 to 10 minutes at the most in a questionnaire. The following always applies: the shorter the questionnaire, the fewer the dropouts, and thus the more analyzable the data. It is often hard to believe how many students overlook this aspect when creating questionnaires: I have seen questionnaires with a length of 10 or more A4 pages, which would easily have taken 30, 45, or even 60 minutes or more to answer — and I doubt very much that the respective students were able to evaluate more than just a handful of fully completed questionnaires in the end. Unrealistic statements about the duration of the survey should also be avoided: Anyone who promises on the survey start page that filling out the survey will take “a maximum of 5 minutes” and then serves 50 individual questions must expect many frustrated dropouts.

Solution: Basically, you should keep every survey as short as possible and only ask what you really need to work on the respective research questions. Each additional question increases the risk that respondents will drop out of the survey due to lack of time or frustration and a high dropout rate, in turn, calls into question the overall validity of the results.

4. Incorrectly selected scale level.

Rate your satisfaction with the product on a school grade scale from 1 (very good) to 6 (unsatisfactory).

What’s wrong with this question? Basically, nothing until the student tells me after the survey has been carried out that the answers are to be included in an analysis of variance. However, only metrically scaled characteristics are allowed and since school grades are ordinally scaled, the intended analysis cannot be carried out with the collected data if only one had thought about this beforehand. In practice, such errors, unfortunately, occur frequently, since the development of the questions and the preparation of an evaluation plan are often mistakenly carried out separately from each other or even worse in teams even by different people. In fact, however, one depends on the other, which is why one should always be clear when formulating the questions about which statistical methods one would like to use later to evaluate the answers.

Solution: The best thing to do is to note down the evaluation procedure desired against the background of the research question(s) (i.e. which mean values are to be calculated, which graphs are to be created, which analyses are to be carried out?) next to the draft questions during the formulation of the questionnaire and to check critically in each case whether the intended calculation will be possible at all with the expected data not only with regard to the scale level.

5. Too many general formulations

What do you think about environmental protection?

a) I think it is rather important.
b) I find it less important.

The frame of reference for this question remains unclear: Do we want to ask the respondents’ opinion on environmental protection in their own living environment, on environmental protection in industry or on the government’s environmental protection policy? The two highly subjective answer options “rather important” and “less important” are similarly obscure. Just as with the double question, it is therefore impossible to know which frame of reference the respondents have chosen and thus which question they have answered at all.

Solution: Questions should always be formulated as precisely as possible as one of the greatest challenges in questionnaire design. Here, too, a pre-test and the subsequent conversation with the test subjects help: Did everyone interpret the question identically?

Which typical formulation errors in questionnaires are still missing? We would like to know more from you.