Lecture 1. Introduction to questionnaires
Questionnaires are used by a broad range of groups with different aims. Examples:
- Governments:
o To measure the health of their nations or economic wellbeing
o To inform policy decisions
- Organizations:
o To do research, investigate the difference in salaries between women and men
(SCP)
o To measure work safety (TNO)
- Companies:
o To select suitable candidates for a job
o To test the usability of a product
o To measure customers’ or members’ satisfaction
o For marketing or evaluation purposes
- Researchers and social scientists:
o Predict voting behaviors or measure political attitudes, quality of life,
personality, behaviors.
What is a questionnaire?
- Simple definition (Merriam Webster dictionary): A written set of questions that are
given to people to collect facts or opinions about something (typical performance)
- The set of questions is called questionnaire (or test) items.
- Questionnaire is usually about typical behavior (someone’s general behavior or opinion,
answers can be given without very specific context) rather than how behavior and
attitudes change in different contexts.
- Questionnaires are also used to see how behaviors change in different contexts, but a
more common way of using questionnaire is about the general opinion or behavior of
people.
- It is one of the main data collection tools in social sciences.
What is an item?
- Item = statement/question + response alternatives (an item consists of both the
question and the answer)
What does a questionnaire (or a test) do?
, - A questionnaire does not try to capture a person with all his/her facets and
complexities. Instead, it tries to isolate 1 characteristic and tries to quantify to what
extent someone has that characteristic.
- Example: a personality questionnaire tries to measure certain aspects of personality. For
example, how extravert is someone, how introvert is someone, how open is someone,
how agreeable is someone. It does not measure all different aspects of personality.
- So, questionnaire gives an incomplete description of reality on purpose. We focus on
certain things on purpose to make our questionnaires more reliable and valid.
Questionnaire consists of different scales
- Example: A questionnaire measuring employee satisfaction. Employee satisfaction
depends on whether you are interested in the tasks you are doing in the day and the
atmosphere of work.
There are different aspects of employee satisfaction => To measure these different
aspects, we use different scales (different set of items). The scale is made up of
many items that specifically measure the specific part of what we want to measure.
o Scale 1: Measure satisfaction with work tasks (items 1 – 20 of the questionnaire)
o Scale 2: Measure work atmosphere (items 21 – 40 of the questionnaire)
Why do we use questionnaires?
- Questionnaires act as a measurement instrument to translate non-observable
psychological characteristics into scores and numbers.
- Questionnaires are used to answer research questions and test hypotheses.
- In order to answer research questions and test hypotheses, we have to measure
constructs that are not directly observable. These constructs have to be operationalized
and made measurable. To make the non-directly-measurable constructs measurable, we
can use questionnaires.
Measurement in social sciences is complex
- In exact sciences, measurement error is low because we can directly observe and
measure the exact and actual amount of something (amount of water, a person’s
height, e.g.). We can directly calculate the measurement error.
- In social sciences, what we want to measure is non-directly observable, and there is no
true value, so we have to make an approximation. (Example: the intelligence of a
potential employee, the quality of life in different countries, the working culture in
organizations). We don’t know how intelligent someone really is or how suitable a
, potential employee is before they start working for us. So, we cannot directly calculate
the measurement error.
- So, a questionnaire is an approximation of something that we cannot measure without
any error.
Many important decisions are made based on questionnaires, so it is important that our
questionnaires have good quality: reliable and valid.
Reliability: the precision of the measurement
- Is there a lot of measurement error? (Are we measuring in a consistent way?) (If Klaas
has a high IQ score based on test A, then we expect that he will have a similarly high IQ
score when tested at a different moment with the same test).
- It reveals how much of the score is real and how much is due to measurement error.
- Compared to the exact sciences where we know the actual and true values (the amount
of water in the bin), this remains an approximation because we don’t know the real IQ
(because a part of the score is due to measurement error)
Validity:
- Are the differences that emerge from the questionnaire really differences in the
construct we are interested in? Example: are the difference IQ scores based on real
differences in IQ? Does the variance in the scores represent real individual differences?
- Is intelligence unidimensional or multidimensional (mathematical ability, reading ability,
e.g.)
Example: Using a measurement tape to measure a person’s height is reliable because we get
the same results every time we measure if we use the tape in the right way. But using the
scores of height to say anything about his or her intelligence is reliable but not valid because we
are using it for a different aim.
Some features of questionnaires
- Content (ask about):
o Behavior: what people do
o Beliefs: what people believe to be true
o Opinions: what people think is desirable
, o Demographic information: characteristics of respondents (age, income, type of
job, education, e.g.). It helps answer the research question and also says
something about the people who don’t fill in or drop out of the questionnaire.
We can say if the group that drops out is a selective group of respondents. For
example: you find out that the older group of respondents drop out. That has an
impact on the results because you are missing a significantly large group of
respondents, so the results can be biased (it does not represent the beliefs or
values of a whole generation anymore).
- Administration mode:
o Interactional mode: with interviewer or self-administered
o Technological modes: telephone, computer, internet, paper and pencil (CAPI,
PAPI, CATI, WAPI, CASI, SAPI, SASI)
o Mix of administration methods.
- Question type: Open vs. Closed format
- Pros and cons:
o Pro: fast way of obtaining data, cheap, can be used for large samples. Very
efficient to reach many people without taking much time.
o Cons: low response rates (especially for mail and internet surveys) => not make
too long questionnaire, answer to questions may deviate from actual
facts/behaviors (satisficing, social desirability, e.g.)
Survey
- A technical/research method for collecting information. It considers all the choices to be
made when answering the research question
o How to collect data
o What the sample should consist of (selection criteria)
o Research methods
- Characteristics:
o Structured or systematic set of data (variable x case data grid)
o Seeking an understanding of what may cause a phenomenon
- Information may be collected using a questionnaire but may also be obtained using an
interview or observation.
- Questionnaire, on the other hand, is just an instrument to collect data.
Survey ≠ Questionnaire.
Test
Questionnaires are used by a broad range of groups with different aims. Examples:
- Governments:
o To measure the health of their nations or economic wellbeing
o To inform policy decisions
- Organizations:
o To do research, investigate the difference in salaries between women and men
(SCP)
o To measure work safety (TNO)
- Companies:
o To select suitable candidates for a job
o To test the usability of a product
o To measure customers’ or members’ satisfaction
o For marketing or evaluation purposes
- Researchers and social scientists:
o Predict voting behaviors or measure political attitudes, quality of life,
personality, behaviors.
What is a questionnaire?
- Simple definition (Merriam Webster dictionary): A written set of questions that are
given to people to collect facts or opinions about something (typical performance)
- The set of questions is called questionnaire (or test) items.
- Questionnaire is usually about typical behavior (someone’s general behavior or opinion,
answers can be given without very specific context) rather than how behavior and
attitudes change in different contexts.
- Questionnaires are also used to see how behaviors change in different contexts, but a
more common way of using questionnaire is about the general opinion or behavior of
people.
- It is one of the main data collection tools in social sciences.
What is an item?
- Item = statement/question + response alternatives (an item consists of both the
question and the answer)
What does a questionnaire (or a test) do?
, - A questionnaire does not try to capture a person with all his/her facets and
complexities. Instead, it tries to isolate 1 characteristic and tries to quantify to what
extent someone has that characteristic.
- Example: a personality questionnaire tries to measure certain aspects of personality. For
example, how extravert is someone, how introvert is someone, how open is someone,
how agreeable is someone. It does not measure all different aspects of personality.
- So, questionnaire gives an incomplete description of reality on purpose. We focus on
certain things on purpose to make our questionnaires more reliable and valid.
Questionnaire consists of different scales
- Example: A questionnaire measuring employee satisfaction. Employee satisfaction
depends on whether you are interested in the tasks you are doing in the day and the
atmosphere of work.
There are different aspects of employee satisfaction => To measure these different
aspects, we use different scales (different set of items). The scale is made up of
many items that specifically measure the specific part of what we want to measure.
o Scale 1: Measure satisfaction with work tasks (items 1 – 20 of the questionnaire)
o Scale 2: Measure work atmosphere (items 21 – 40 of the questionnaire)
Why do we use questionnaires?
- Questionnaires act as a measurement instrument to translate non-observable
psychological characteristics into scores and numbers.
- Questionnaires are used to answer research questions and test hypotheses.
- In order to answer research questions and test hypotheses, we have to measure
constructs that are not directly observable. These constructs have to be operationalized
and made measurable. To make the non-directly-measurable constructs measurable, we
can use questionnaires.
Measurement in social sciences is complex
- In exact sciences, measurement error is low because we can directly observe and
measure the exact and actual amount of something (amount of water, a person’s
height, e.g.). We can directly calculate the measurement error.
- In social sciences, what we want to measure is non-directly observable, and there is no
true value, so we have to make an approximation. (Example: the intelligence of a
potential employee, the quality of life in different countries, the working culture in
organizations). We don’t know how intelligent someone really is or how suitable a
, potential employee is before they start working for us. So, we cannot directly calculate
the measurement error.
- So, a questionnaire is an approximation of something that we cannot measure without
any error.
Many important decisions are made based on questionnaires, so it is important that our
questionnaires have good quality: reliable and valid.
Reliability: the precision of the measurement
- Is there a lot of measurement error? (Are we measuring in a consistent way?) (If Klaas
has a high IQ score based on test A, then we expect that he will have a similarly high IQ
score when tested at a different moment with the same test).
- It reveals how much of the score is real and how much is due to measurement error.
- Compared to the exact sciences where we know the actual and true values (the amount
of water in the bin), this remains an approximation because we don’t know the real IQ
(because a part of the score is due to measurement error)
Validity:
- Are the differences that emerge from the questionnaire really differences in the
construct we are interested in? Example: are the difference IQ scores based on real
differences in IQ? Does the variance in the scores represent real individual differences?
- Is intelligence unidimensional or multidimensional (mathematical ability, reading ability,
e.g.)
Example: Using a measurement tape to measure a person’s height is reliable because we get
the same results every time we measure if we use the tape in the right way. But using the
scores of height to say anything about his or her intelligence is reliable but not valid because we
are using it for a different aim.
Some features of questionnaires
- Content (ask about):
o Behavior: what people do
o Beliefs: what people believe to be true
o Opinions: what people think is desirable
, o Demographic information: characteristics of respondents (age, income, type of
job, education, e.g.). It helps answer the research question and also says
something about the people who don’t fill in or drop out of the questionnaire.
We can say if the group that drops out is a selective group of respondents. For
example: you find out that the older group of respondents drop out. That has an
impact on the results because you are missing a significantly large group of
respondents, so the results can be biased (it does not represent the beliefs or
values of a whole generation anymore).
- Administration mode:
o Interactional mode: with interviewer or self-administered
o Technological modes: telephone, computer, internet, paper and pencil (CAPI,
PAPI, CATI, WAPI, CASI, SAPI, SASI)
o Mix of administration methods.
- Question type: Open vs. Closed format
- Pros and cons:
o Pro: fast way of obtaining data, cheap, can be used for large samples. Very
efficient to reach many people without taking much time.
o Cons: low response rates (especially for mail and internet surveys) => not make
too long questionnaire, answer to questions may deviate from actual
facts/behaviors (satisficing, social desirability, e.g.)
Survey
- A technical/research method for collecting information. It considers all the choices to be
made when answering the research question
o How to collect data
o What the sample should consist of (selection criteria)
o Research methods
- Characteristics:
o Structured or systematic set of data (variable x case data grid)
o Seeking an understanding of what may cause a phenomenon
- Information may be collected using a questionnaire but may also be obtained using an
interview or observation.
- Questionnaire, on the other hand, is just an instrument to collect data.
Survey ≠ Questionnaire.
Test
