Lecture 1: Introduction to questionnaires
Questionnaires are used by:
• Governments
• to measure the health of their nations or their economic wellbeing
• To inform policy decisions
• Organisations:
• To investigate the difference in salaries between women and men (SCP,
“emancipation monitor 2018”)
• Work safety (TNO)
• Companies:
• To select suitable candidates for a job
• To test the “usability” of a product (e.g., Philips)
• To measure customers’ or members’ satisfaction
• for marketing or evaluation purposes
• Researchers social sciences
• Predict voting behaviors or measure political attitudes, measure quality of life,
personality or behavior..
Questionnaire: a written set of questions that are given to people in order to collect facts or
opinions about something” (typical performance)
- Set of questions: Questionnaire (or test) items
- Typical behavior, often questions that can be answered without very specific context (so
generally not how behavior and attitudes can change in a different context, but rather
someone’s general behavior or opinion)
- One of the main data collection tools in the social sciences (and in empirical research in
general, also in other disciplines)
Item: statement/question + response alternatives
Statement: When I know that the dentist is going to extract a tooth I am already afraid in the
waiting room.
Response alternatives:
1 2 3 4 5
Completely not … … … Completely
applicable to me applicable to me
One measurement instrument (questionnaire or test) does not try to capture a person with al
his/her facets and complexities. It tries to isolate 1 characteristic and tries to quantify to what extent
someone “has that” e.g. how introvert is someone.
- It gives an incomplete description of reality on purpose
Questionnaires often consist of different scales:
- E.g.: questionnaire to measure employee satisfaction:
o Scale 1: Measures work atmosphere (items 1-20 of the questionnaire)
o Scale 2: Measures satisfaction with work tasks (item 21-40 of the questionnaire)
These different scales measure different aspects of employee satisfaction.
The respondents differ in employee satisfaction. Like this
we measured a characteristic which was not directly
observable on first hand.
1
,In the end, we are interested in answering research questions and testing hypotheses
Example:
“Ignorance is bliss” That is: “Intelligent people are more often depressed”
• Are more intelligent people more often depressed?
• How can we measure depression, how intelligence? Both are not directly observable; that
is we cannot just use a measurement tape and keep it next to someone’s brain..
=> Constructs that we cannot directly observe have to be operationalized and made
measurable. For the second part (measuring) we can use questionnaires.
Hypothesis: Intelligent people are more often depressed
(‘Ignorance is bliss’)
We can test this by looking at the relationship between
intelligence and depression (prediction), e.g.:
“The higher someone’s IQ is, the more likely he or she
has depression symptoms”
-> Measuring intelligence & depression:
-> Test / questionnaire
So what we need is:
• 1) A sample of people willing to participate in our research
• 2) A way to measure intelligence and depression (well, simply a test and a
questionnaire, right?).
Easy!!! Right? …
Measurement in the social sciences
The measurement of physical traits like for example someone’s length is not that hard:
- If I and someone from your group measures my length, then we will roughly (give or take a
few mm) have the same result; or with other words: there is not a lot of measurement error.
Why? We can directly observe the trait.
However, in the social sciences, we are interested in traits that are not directly observable.
- E.g., the intelligence of a potential employee (HR) or the quality of life in different countries
(sociology) or the working culture in different organizations (organization sciences)
In the exact sciences, we often can directly measure something (e.g., quality of a measurement
instrument that measures the amount of water in a bin, where we know the exact and actual
(“true”) amount of water)
In the social sciences, there is no “true” value”, so we often have to make an approximation (e.g., we
don’t know how intelligent someone really is, or how suitable a potential employee is before he/she
actually starts working for us)
So you could say that a questionnaire is really just an approximation of something that we cannot
measure without any error.
However, still, many very important decisions are made based on questionnaires!! (e.g., selection of
candidates, continuation of therapy, whether a child is dyslectic, etc…)
And therefore it is of utter importance that our questionnaires are of good quality:
• Reliable
• Valid
2
,Reliability is about the precision of the measurement
- Is there a lot of measurement error? Which means; are we measuring in a consistent way? If
Klaas has a high IQ score based on test A, then we expect that he will have a high (similar) IQ
score when tested at a different moment with the same test
Compared to the exact sciences where we know the actual (“true” values) this remains an
approximation, because we don’t know the “real” IQ.
Validity: is our questionnaire really measuring what we would like to measure? Are the differences
that evolve from the questionnaire really differences in the construct we are interested in?
- E.g., are IQ scores that are different based on real differences in IQ?
- Is intelligence unidimensional or are there multiple dimensions to it? (e.g., mathematical
ability, reading ability..)
Features of questionnaires:
- Questionnaires can ask about (content)
o Behavior, what people do (e.g., questioning mothers with children at school age: do
you work?)
o Beliefs, wat people believe to be true/false (e.g., how many percent of mothers with
children at school-age do you think work)
o Opinions/values/attitudes: What do people think is desirable?
o Demographical information: obtain information about characteristics of the
respondent (age, income, type of job)
- Administration mode
o Interactional mode: With interviewer or self-administered
o Technological mode: Telephone, computer, internet, paper-and-pencil
o Mix of administration methods
- Question type: Open versus closed format
Pros and cons
Pro:
- Questionnaires are a fast way of obtaining data, cheap
- Can be used for large samples
Con:
- Low response rates (especially for mail and internet surveys)
- Given answer may deviate from actual facts/behavior (e.g. satisficing, social desirability)
Survey: a technique/ research method for collecting information characterized by 1) structured or
systematic set of data (variable × case data grid) 2) seeking an understanding of what may cause a
phenomenon. Information may be collected using a questionnaire but may also be obtained using an
interview or observation for example. So: survey ≠ questionnaire
Test: Mainly understood as an achievement test. Often in the form of a series
of questions where the answers are qualified in terms of ‘right’ or ‘wrong’ →
- Ask about someone’s maximal performance ≠ questionnaires ask
about typical performance
• May be based on / supplemented with tasks
- e.g. WAIS (Wechsler Adult Intelligence Scale)
3
, Lecture 2: How to make constructs measurable
Questionnaire construction is important for making non-observable constructs measurable.
Measurement in the social sciences
In ‘technical’ studies measurement is relatively easy (e.g., measuring the amount of fluids, etc.) and
per definition (relatively) reliable.
Measurement in the social sciences: Often, we are interested in measuring a trait or behavior. For
example, employee satisfaction, intelligence… these are not directly observable (“latent”) and as
such not directly measurable
- This is difficult and can be arbitrary because person A might define such a trait different than
person B. Thus often per definition not reliable (if we don’t pay attention to the statistics)
Constructs: unobservable concepts and therefore we cannot directly measure them. In the scientific
literature: “latent”, psychological traits are often referred to as “latent traits” e.g. intelligence.
Constructs in social science research
Social science research is often based on relations between constructs:
- Countries with more poor people / people living below the poverty line have lower calory
intakes. (poverty line is already less abstract than ‘poor’)
- If organizational culture is not aligned with the new organizational strategy, the
implementation of the new strategy will not be successful.
- Extravert people make successful managers.
- Increasing crime rates are caused by a lack of social capital.
These all include constructs that we can’t directly observe and thus can’t directly measure.
Constructs are mental representations and are typically based on experience; abstract summaries of
a whole set of characteristics, behaviors, attitudes we see as having something in common
- Problem: Most constructs cannot be measured in a direct way (contrary to hair color, sex,
length, bmi, …) but are latent e.g. intelligence, poverty, social capital, …
Solution: Indirect measurement by using a questionnaire. But before, we have to operationalize the
(abstract) construct.
1. Define what we mean by the concept. Develop our own nominal definition (e.g., after
literature search, checking encyclopedia, web)
2. What are the different aspects (dimensions) of the construct? Dimensions may be further
subdivided in sub-dimensions (and further, and further …)
3. Until we get to the point where we have aspects of the construct that are directly
measurable-> operational definition
Example 1 operationalizing
Social capital (e.g. Increasing crime rates are caused by a lack of social capital.)
- We start with the definition of the construct. However, this can already be hard. Different
researchers might use different definitions because these are abstract things.
Definition: “Social capital consists of networks of social relations which are characterized by norms of
trust and reciprocity. Combined, it is these elements which are argued to sustain civil society and
which enable people to act for mutual benefit” (Stone, 2001)
When we look at social capital, we see 2 dimensions:
- Structure of social relationships – networks
- Quality of social relationships - norms
We can further divide these to get to something which is less and less abstract. Sub-dimensions of
Networks and Norms dimensions:
- Networks:
o Type
4
Questionnaires are used by:
• Governments
• to measure the health of their nations or their economic wellbeing
• To inform policy decisions
• Organisations:
• To investigate the difference in salaries between women and men (SCP,
“emancipation monitor 2018”)
• Work safety (TNO)
• Companies:
• To select suitable candidates for a job
• To test the “usability” of a product (e.g., Philips)
• To measure customers’ or members’ satisfaction
• for marketing or evaluation purposes
• Researchers social sciences
• Predict voting behaviors or measure political attitudes, measure quality of life,
personality or behavior..
Questionnaire: a written set of questions that are given to people in order to collect facts or
opinions about something” (typical performance)
- Set of questions: Questionnaire (or test) items
- Typical behavior, often questions that can be answered without very specific context (so
generally not how behavior and attitudes can change in a different context, but rather
someone’s general behavior or opinion)
- One of the main data collection tools in the social sciences (and in empirical research in
general, also in other disciplines)
Item: statement/question + response alternatives
Statement: When I know that the dentist is going to extract a tooth I am already afraid in the
waiting room.
Response alternatives:
1 2 3 4 5
Completely not … … … Completely
applicable to me applicable to me
One measurement instrument (questionnaire or test) does not try to capture a person with al
his/her facets and complexities. It tries to isolate 1 characteristic and tries to quantify to what extent
someone “has that” e.g. how introvert is someone.
- It gives an incomplete description of reality on purpose
Questionnaires often consist of different scales:
- E.g.: questionnaire to measure employee satisfaction:
o Scale 1: Measures work atmosphere (items 1-20 of the questionnaire)
o Scale 2: Measures satisfaction with work tasks (item 21-40 of the questionnaire)
These different scales measure different aspects of employee satisfaction.
The respondents differ in employee satisfaction. Like this
we measured a characteristic which was not directly
observable on first hand.
1
,In the end, we are interested in answering research questions and testing hypotheses
Example:
“Ignorance is bliss” That is: “Intelligent people are more often depressed”
• Are more intelligent people more often depressed?
• How can we measure depression, how intelligence? Both are not directly observable; that
is we cannot just use a measurement tape and keep it next to someone’s brain..
=> Constructs that we cannot directly observe have to be operationalized and made
measurable. For the second part (measuring) we can use questionnaires.
Hypothesis: Intelligent people are more often depressed
(‘Ignorance is bliss’)
We can test this by looking at the relationship between
intelligence and depression (prediction), e.g.:
“The higher someone’s IQ is, the more likely he or she
has depression symptoms”
-> Measuring intelligence & depression:
-> Test / questionnaire
So what we need is:
• 1) A sample of people willing to participate in our research
• 2) A way to measure intelligence and depression (well, simply a test and a
questionnaire, right?).
Easy!!! Right? …
Measurement in the social sciences
The measurement of physical traits like for example someone’s length is not that hard:
- If I and someone from your group measures my length, then we will roughly (give or take a
few mm) have the same result; or with other words: there is not a lot of measurement error.
Why? We can directly observe the trait.
However, in the social sciences, we are interested in traits that are not directly observable.
- E.g., the intelligence of a potential employee (HR) or the quality of life in different countries
(sociology) or the working culture in different organizations (organization sciences)
In the exact sciences, we often can directly measure something (e.g., quality of a measurement
instrument that measures the amount of water in a bin, where we know the exact and actual
(“true”) amount of water)
In the social sciences, there is no “true” value”, so we often have to make an approximation (e.g., we
don’t know how intelligent someone really is, or how suitable a potential employee is before he/she
actually starts working for us)
So you could say that a questionnaire is really just an approximation of something that we cannot
measure without any error.
However, still, many very important decisions are made based on questionnaires!! (e.g., selection of
candidates, continuation of therapy, whether a child is dyslectic, etc…)
And therefore it is of utter importance that our questionnaires are of good quality:
• Reliable
• Valid
2
,Reliability is about the precision of the measurement
- Is there a lot of measurement error? Which means; are we measuring in a consistent way? If
Klaas has a high IQ score based on test A, then we expect that he will have a high (similar) IQ
score when tested at a different moment with the same test
Compared to the exact sciences where we know the actual (“true” values) this remains an
approximation, because we don’t know the “real” IQ.
Validity: is our questionnaire really measuring what we would like to measure? Are the differences
that evolve from the questionnaire really differences in the construct we are interested in?
- E.g., are IQ scores that are different based on real differences in IQ?
- Is intelligence unidimensional or are there multiple dimensions to it? (e.g., mathematical
ability, reading ability..)
Features of questionnaires:
- Questionnaires can ask about (content)
o Behavior, what people do (e.g., questioning mothers with children at school age: do
you work?)
o Beliefs, wat people believe to be true/false (e.g., how many percent of mothers with
children at school-age do you think work)
o Opinions/values/attitudes: What do people think is desirable?
o Demographical information: obtain information about characteristics of the
respondent (age, income, type of job)
- Administration mode
o Interactional mode: With interviewer or self-administered
o Technological mode: Telephone, computer, internet, paper-and-pencil
o Mix of administration methods
- Question type: Open versus closed format
Pros and cons
Pro:
- Questionnaires are a fast way of obtaining data, cheap
- Can be used for large samples
Con:
- Low response rates (especially for mail and internet surveys)
- Given answer may deviate from actual facts/behavior (e.g. satisficing, social desirability)
Survey: a technique/ research method for collecting information characterized by 1) structured or
systematic set of data (variable × case data grid) 2) seeking an understanding of what may cause a
phenomenon. Information may be collected using a questionnaire but may also be obtained using an
interview or observation for example. So: survey ≠ questionnaire
Test: Mainly understood as an achievement test. Often in the form of a series
of questions where the answers are qualified in terms of ‘right’ or ‘wrong’ →
- Ask about someone’s maximal performance ≠ questionnaires ask
about typical performance
• May be based on / supplemented with tasks
- e.g. WAIS (Wechsler Adult Intelligence Scale)
3
, Lecture 2: How to make constructs measurable
Questionnaire construction is important for making non-observable constructs measurable.
Measurement in the social sciences
In ‘technical’ studies measurement is relatively easy (e.g., measuring the amount of fluids, etc.) and
per definition (relatively) reliable.
Measurement in the social sciences: Often, we are interested in measuring a trait or behavior. For
example, employee satisfaction, intelligence… these are not directly observable (“latent”) and as
such not directly measurable
- This is difficult and can be arbitrary because person A might define such a trait different than
person B. Thus often per definition not reliable (if we don’t pay attention to the statistics)
Constructs: unobservable concepts and therefore we cannot directly measure them. In the scientific
literature: “latent”, psychological traits are often referred to as “latent traits” e.g. intelligence.
Constructs in social science research
Social science research is often based on relations between constructs:
- Countries with more poor people / people living below the poverty line have lower calory
intakes. (poverty line is already less abstract than ‘poor’)
- If organizational culture is not aligned with the new organizational strategy, the
implementation of the new strategy will not be successful.
- Extravert people make successful managers.
- Increasing crime rates are caused by a lack of social capital.
These all include constructs that we can’t directly observe and thus can’t directly measure.
Constructs are mental representations and are typically based on experience; abstract summaries of
a whole set of characteristics, behaviors, attitudes we see as having something in common
- Problem: Most constructs cannot be measured in a direct way (contrary to hair color, sex,
length, bmi, …) but are latent e.g. intelligence, poverty, social capital, …
Solution: Indirect measurement by using a questionnaire. But before, we have to operationalize the
(abstract) construct.
1. Define what we mean by the concept. Develop our own nominal definition (e.g., after
literature search, checking encyclopedia, web)
2. What are the different aspects (dimensions) of the construct? Dimensions may be further
subdivided in sub-dimensions (and further, and further …)
3. Until we get to the point where we have aspects of the construct that are directly
measurable-> operational definition
Example 1 operationalizing
Social capital (e.g. Increasing crime rates are caused by a lack of social capital.)
- We start with the definition of the construct. However, this can already be hard. Different
researchers might use different definitions because these are abstract things.
Definition: “Social capital consists of networks of social relations which are characterized by norms of
trust and reciprocity. Combined, it is these elements which are argued to sustain civil society and
which enable people to act for mutual benefit” (Stone, 2001)
When we look at social capital, we see 2 dimensions:
- Structure of social relationships – networks
- Quality of social relationships - norms
We can further divide these to get to something which is less and less abstract. Sub-dimensions of
Networks and Norms dimensions:
- Networks:
o Type
4
