7 Introduction to Questionnaire Design

Author: Trang Bui

Last Updated: Nov 17, 2020

Surveys or questionnaires play an important role in many research and studies. A well-designed questionnaire will help researchers collect the data required for their research questions with strong validity and reliability. This section aims to provide some guidelines to create such questionnaires.

The most important thing to keep in mind when designing a questionnaire is that we are asking the respondents a favor. Therefore, the questionnaire needs to be as easy and less time-consuming as possible, while gathering enough information for the research questions. The goal is to create a short, easy, inviting but still sound and logical questionnaire.

There are four main steps to achieve our goal:

7.1 Plan the Study

Planning the study ahead allows us to create a questionnaire that meets the research goal while keeping it short and concise. In the questionnaire, we want to include questions that provide enough, but not more than what we need. In this section, suppose we are interested to create a survey to investigate students’ mental health. To plan the study, we should consider the following:

What is the topic of interest?

The topic based on our example is the mental health among students.
What is the target population, i.e. the population of interest and to draw conclusion on?

Suppose researchers want to make conclusions on the mental health of students at the University of Waterloo exclusively, instead of, for example, the mental health of students in Ontario province. Then, the target population is all the students at the University of Waterloo.
To whom or to which groups do we want to report to?

Suppose researchers want to report or compare the results of students’ mental health based on gender, faculty, country of origin, etc.
What are the research questions, i.e. hypotheses?

Is stress level related to the students’ faculty? What are the common mental health problems among students?
What analysis tools will be used after the data has been collected?

R, SAS, STATA, etc.

Examining as many aspects of the study as possible will paint a picture of what the questionnaire should include, what questions need to be asked, and to whom the questionnaire should be designed for and sent to, etc. It is helpful to write down the plan for future reference and further discussion.

7.2 Prepare the Questions

In this step, we design the questions based on the research plan laid out. The many aspects of a question that need to be considered include, but are not limited to, the type of question, the wording of the questions, the answer options for the questions, the order of the questions, etc. All of these will affect the difficulty of the questionnaire, the respondents’ willingness to answer or to answer correctly and truthfully, and also the analysis that will be carried out.

When creating a question, think about the thought process that the survey respondents have to go through to answer. We also need to find ways to help them provide responses that are in line with the research goals. Here are a few questions to consider when creating each question:

How can we define or decompose certain concepts? Is it fair to assume that all the respondents understand the wordings or notions? Is there any chance that the respondents might misunderstand the definitions, wordings, or the question itself?

“Have you ever had depression?”

The term “depression” may not be familiar to the respondents. Can everyone in the target population differentiate between depression and sadness? To avoid misunderstanding, the symptoms of depression can be listed before asking the question.
How long ago since the event of interest occurred? Is it easy for the respondents to recall and retrieve this information? Is there any cue in the survey or record the respondents can refer to?

“When was the last time that you met a psychiatrist?”.

For some people, it may be so long ago that they cannot remember. Asking them to have their appointment record easily accessible can help with retrieving this information.
Do the respondents need to estimate or judge to obtain the answer? What can be some references for these estimates or judgments?

“How many hours do you spend on self-care per week?”.

Suppose a person spends three hours a day, the estimate would be 21 hours a week. However, it can be difficult for the person to recall the actual time length. To assist the respondents, provide some time point reference in the question. For example, time spent on self-care when they wake up or before going to bed. Both morning and bedtime routines are time points that respondents can remember easily.
Can the respondents easily map their answers to the options provided? Are all the possible answers included in the question’s answer categories? How can we help the respondents express their answers precisely?

“What do you think are the reasons for your sadness? (Select all that apply.)”

Although the respondents can choose as many options as possible for the question, they may still not be able to express all of their answers.

“How do you rate your happiness?”

We can use a scale of 1-10, where 1 indicates “very unhappy” and 10 indicates “very happy”. A better choice could be a scale that ranges from -5 to 5, where -5 implies “very unhappy”, 0 corresponds to “indifferent”, and 5 as “very happy”. This scale will be easier to relate to.

The context of the question can affect the respondents’ thought process and hence, their answers. It is important to make sure that the context is expressed or explained clearly. This can be achieved by

Asking the general question before the specific questions.

Q1: “How do you feel about your performance at school?”

Q2: “How do you rate your level of happiness?”

When we asked about how the respondents feel about their performance at school first, they may associate happiness with performance at school, and ignore their happiness that is not associated with school performance when answering Q2. If the order of the questions were flipped, the respondents would be more likely to rate their happiness level based on all aspects of their life instead of focusing on school performance.
Grouping related questions in modules.

The order of the answer options can also affect the validity of the answers. Respondents tend to choose the first or last few options when there are many options. Consider using the natural order of the answer options such as alphabetical order. If the natural order is not available, and the list of options is long, randomize the order of the options.

It is challenging to create sensitive questions that respondents are willing to answer and to answer truthfully. Respondents usually are hesitant to answer because of privacy and (or) shame.

If the concern is privacy, we can create questions that do not require exact information, but an estimate.

How old are you?

Instead of asking for the exact age, we can ask respondents to choose from the age group they belong to.
If the concern is shame, or social desirability, consider ways that will ease the respondents to the answer.

Do you take anti-depressant?

To help respondents feel more comfortable to provide a truthful answer to the question, we can add a sentence or two about the benefits of anti-depressant on improving moods, sleep quality, etc.

7.3 Put Together the Questionnaire

In the beginning, provide an introduction about the research topic and its importance to motivate the respondents to answer the questionnaire. Mention the research organizations, investigators, or sponsors and an estimate of the completion time of the questionnaire. Do not forget to include statements about privacy and confidentiality from the Office of Research Ethics. Including relevant graphics may appeal to the respondents that this is an easy-to-answer survey.

The information obtained from this survey will help the university better understand students’ mental well-being and take reasonable measures to help the students.
Start with some questions that are related to the topic and easy to answer. Questions with “Yes-No” answers will usually ease the respondents to the questionnaire.
Group the questions into modules or sections to create a logical flow that is easy for respondents to follow. In between these modules, a short description of the module can be added to make the transitions smoother.
Ask sensitive questions near the end of the survey when the respondents are comfortable to provide their answers or opinions. We also recommend asking only the demographic questions required for the research questions at the end of the questionnaire. It may be worthy to mention that personal information will only be used for statistical purposes.
Finally, give the respondents a big thank you. If there are any incentives for completing the survey, collect the required information. Additionally, again strongly stress the privacy of the gathered information. It is also a good place to leave the researchers’ contact information in the event that the respondents have any concerns or want to give additional feedback.

7.4 Pretest the Questionnaire

It is recommended to pretest the questionnaire before conducting data collection. This will help identify problems in the questionnaire and allow for revision and improvement.

Consider pretesting the questionnaire by:

obtaining expert review from colleagues who have the same or similar research topics, or statistical consultants who have expertise in data collection and analysis;
conducting interviews with focus groups, i.e. a small group in the target population. This method allows us to know whether the prospective respondents have any difficulty in understanding and answering the questions. Their feedback or advice can be used to revise the questionnaire.

Designing questionnaires is an iterative process. The four steps mentioned above can be revisited and revised as needed. If you have any further questions about questionnaires design, please consult a consultant at the Statistical Consulting and Collaborative Research, or the Survey Research Centre.

7.5 A Checklist for Questionnaire Designs

Planning
- What is the topic of interest?
- What is the target population?
- To whom or to which groups do you want to report?
- What are the research questions?
- What analysis tools will be used for data analysis?
Preparing the questions
- Can the respondents understand the concepts in the question?
- Can the respondents easily recall the information required to answer the question?
- How will the respondents estimate or judge to answer this question?
- Can the respondents easily map their answers to the options provided?
- Will the context of the question affect the respondents’ thought processes?
- If there is a long list of options, is there a natural ordering? Is randomization of order needed?
- Is the question sensitive? How do we ease the respondents into answering the question?
Putting together the questions
- Preamble:
  - Provide an introduction to the research and highlight its importance.
  - List the name of the research organizations, investigators, or sponsors.
  - Estimate the completion time of the questionnaire.
  - Include privacy and confidentiality statements.
- Main body:
  - Start with easy Yes-No questions.
  - Group the questions into modules and create a logical flow.
  - Ask sensitive questions near the end.
  - Ask demographic questions for statistical purposes.
- Ending:
  - Thank the respondents.
  - Ask for contact information if a reward is promised.
  - Leave contacts for further inquiries and feedback.
Pretesting
- Use expert review and (or) focus groups.
- Iterate the steps above until the questionnaire is satisfactory.