SUMMARY CONSTRUCTION AND ANALYSIS OF
QUESTIONNAIRES
LECTURE 1
Questionnaires are used by:
- Governments:
o To measure the health of their nations or their economic wellbeing.
o To make policy decisions.
- Organizations:
o To investigate the difference in salaries between women and men.
o Work safety.
- Companies
o To select suitable candidates for a job.
o To test the “usability” of a product.
o To measure the satisfaction of customers or members.
o For marketing or evaluation purposes.
- Researchers:
o To predict voting behaviors or measure political attitudes, measure
quality of life, personality or behavior.
Questionnaire – a set of printed or written questions with a choice of answers,
devised for the purposes of a survey of statistical study.
Two types of performance measured:
- Maximum performance: what a person can do when motivated?
o Determined most by ability.
o Aptitude tests, achievement tests, intelligence tests.
o Focus on obtaining highest scores.
- Typical performance: what a person will do?
o Determined most by personality.
o Personality questionnaires, behavioral questionnaires.
o Focus on obtaining representative responses (no right or wrong
answers).
Item – statement (or question) and the responsive alternatives.
Purpose of a questionnaire > tries to isolate one characteristic and tries to
quantify to what extent someone ‘has that’. Gives an incomplete description of
reality (on purpose).
Parametric data – data that follows a specific probability distribution, such as
normal distribution.
Questionnaire often consists of different scales (a coherent and often fixed set of
items):
- Example: questionnaire to measure employee satisfaction:
o Scale 1: measures work atmosphere (items 1-20 of the
questionnaire).
o Scale 2: measures satisfaction with work tasks (item 21-40 of the
questionnaire).
,Latent – not directly observable. Cannot be observed and measured directly.
- Example: depression or intelligence. Both not directly observable.
Measurement error
- Example: if several people from your course would measure my height, we
would have the same result > not much measurement error.
Validity – is what we measure truly what we intend to measure?
- Example: bathroom scales (weight of a child vs. weight of a child holding a
dog)? > not valid.
- Example: intelligence measure by ‘asking a friend’? valid.
Reliability – will I get the same result if I measure something repeatedly and
under the same conditions?
- Example: bathroom scales (weight of a child vs. weight of a child holding a
dog)? > reliable
- Example: intelligence measure by ‘asking a friend’? not reliable.
Content of questionnaires:
- Behavior > what people do?
- Beliefs > what people believe to be true/false?
- Opinions/ values/ attitudes > what do people believe is desirable?
- Demographics > get information about the respondent’s characteristics.
Features of questionnaires:
- Mode of questionnaire administration:
o Interactional mode: with interviewer or self-administered.
o Technological mode: telephone, computer, internet etc.
o Mix of administration modes.
- Question type: open-ended versus close-ended questions.
o Open-ended questions – questions that allow someone to give a
free-form answer.
o Close-ended questions – can be answered with a ‘yes’ or ‘no’, or
they have limited set of possible answers.
Pros of a questionnaire
- Very fast and efficient way to collect data.
- Cheap (especially if purchasing is done digitally).
- Suitable for research in large samples.
Cons of a questionnaire
- Low response rate (especially for online surveys).
- Answer to questions may deviate from facts/ behavior (e.g. due to social
desirability).
Survey – technique/ research method for collecting information.
- Structured or systematic set of data.
- Investigating what may be responsible for a particular phenomenon.
It is possible to collect information using a questionnaire, by interviewing
(qualitative surveys or observing > Survey is not the same as a questionnaire!
LECTURE 2
,Constructs – abstract summaries (mental representations) of a number of
characteristics, behaviors, and attitudes that share something in common. Based
on experience.
- Most constructs cannot be measured directly > latent.
Making constructs measurable
1. Use an existing (and possibly already validated) questionnaire.
2. Create your own set of questionnaire items.
o Heuristic approach (‘Descending the ladder of abstraction’).
Define what you mean by the given concept > nominal
definition.
What are the different aspects (dimensions) of the construct?
Dimensions may be further subdivided in sub-dimensions.
Work on this until you have something measurable >
operational definition – a statement that maps one or more
empirical measure onto one or more theoretical constructs.
Example:
1. Nominal definition: ‘Social capital consists of networks of social
relations which are characterized by norms of trust and reciprocity.
Combined, it is these elements which are argued to sustain civil
society, and which enable people to act for mutual benefit’ (Stone,
2001).
2. Aspects/ (sub)dimensions:
Structure of social relationships > networks (type, size and capacity).
Quality of social relationships > norms (trust, reciprocity).
, Established methods of scale construction
1. Construct method (deductive)
o Deductive in nature
Based on a conceptual and
theoretical framework
Guided by testing hypotheses.
Items are derived from a
theoretical definition of the
construct.
o Nomological network - a theoretical
network of associations of the construct
with other variables derived from the
construct theory.
o Item production stage: operational
definition (of all variables).
o Construct validity: convergent and
divergent validity of the items important
for scale construction after a first administration of an item set.
o Aim: a homogenous set of items covering all aspects of the
construct.
How to evaluate? Empirically: MTMM matrix, CFA, reliability analysis.
2. Facet design method.
o Deductive in nature: items are derived from a mapping sentence.
o Facets and facet elements instead of nomological network.
o Content validity: evaluates how well an instrument covers all
relevant parts (the so-called facets) of the construct it aims to
measure dimensionality analysis.
1. Identifying the behavioral features or processes that are
essential to the construct.
2. Essential aspects > facets are specified.
Facets related to the content of the item: situational facets,
behavioral facets.
Facet related to the answer alternatives of the item.
3. For each facet, its elements are defined (called > structs).
4. The facets are completely cross classified (and each combination
of unique structs is called a structuple).
Facets should be independent and mutually exclusive!
Example: dental anxiety questionnaire > structuple (a1, b3, c1).
QUESTIONNAIRES
LECTURE 1
Questionnaires are used by:
- Governments:
o To measure the health of their nations or their economic wellbeing.
o To make policy decisions.
- Organizations:
o To investigate the difference in salaries between women and men.
o Work safety.
- Companies
o To select suitable candidates for a job.
o To test the “usability” of a product.
o To measure the satisfaction of customers or members.
o For marketing or evaluation purposes.
- Researchers:
o To predict voting behaviors or measure political attitudes, measure
quality of life, personality or behavior.
Questionnaire – a set of printed or written questions with a choice of answers,
devised for the purposes of a survey of statistical study.
Two types of performance measured:
- Maximum performance: what a person can do when motivated?
o Determined most by ability.
o Aptitude tests, achievement tests, intelligence tests.
o Focus on obtaining highest scores.
- Typical performance: what a person will do?
o Determined most by personality.
o Personality questionnaires, behavioral questionnaires.
o Focus on obtaining representative responses (no right or wrong
answers).
Item – statement (or question) and the responsive alternatives.
Purpose of a questionnaire > tries to isolate one characteristic and tries to
quantify to what extent someone ‘has that’. Gives an incomplete description of
reality (on purpose).
Parametric data – data that follows a specific probability distribution, such as
normal distribution.
Questionnaire often consists of different scales (a coherent and often fixed set of
items):
- Example: questionnaire to measure employee satisfaction:
o Scale 1: measures work atmosphere (items 1-20 of the
questionnaire).
o Scale 2: measures satisfaction with work tasks (item 21-40 of the
questionnaire).
,Latent – not directly observable. Cannot be observed and measured directly.
- Example: depression or intelligence. Both not directly observable.
Measurement error
- Example: if several people from your course would measure my height, we
would have the same result > not much measurement error.
Validity – is what we measure truly what we intend to measure?
- Example: bathroom scales (weight of a child vs. weight of a child holding a
dog)? > not valid.
- Example: intelligence measure by ‘asking a friend’? valid.
Reliability – will I get the same result if I measure something repeatedly and
under the same conditions?
- Example: bathroom scales (weight of a child vs. weight of a child holding a
dog)? > reliable
- Example: intelligence measure by ‘asking a friend’? not reliable.
Content of questionnaires:
- Behavior > what people do?
- Beliefs > what people believe to be true/false?
- Opinions/ values/ attitudes > what do people believe is desirable?
- Demographics > get information about the respondent’s characteristics.
Features of questionnaires:
- Mode of questionnaire administration:
o Interactional mode: with interviewer or self-administered.
o Technological mode: telephone, computer, internet etc.
o Mix of administration modes.
- Question type: open-ended versus close-ended questions.
o Open-ended questions – questions that allow someone to give a
free-form answer.
o Close-ended questions – can be answered with a ‘yes’ or ‘no’, or
they have limited set of possible answers.
Pros of a questionnaire
- Very fast and efficient way to collect data.
- Cheap (especially if purchasing is done digitally).
- Suitable for research in large samples.
Cons of a questionnaire
- Low response rate (especially for online surveys).
- Answer to questions may deviate from facts/ behavior (e.g. due to social
desirability).
Survey – technique/ research method for collecting information.
- Structured or systematic set of data.
- Investigating what may be responsible for a particular phenomenon.
It is possible to collect information using a questionnaire, by interviewing
(qualitative surveys or observing > Survey is not the same as a questionnaire!
LECTURE 2
,Constructs – abstract summaries (mental representations) of a number of
characteristics, behaviors, and attitudes that share something in common. Based
on experience.
- Most constructs cannot be measured directly > latent.
Making constructs measurable
1. Use an existing (and possibly already validated) questionnaire.
2. Create your own set of questionnaire items.
o Heuristic approach (‘Descending the ladder of abstraction’).
Define what you mean by the given concept > nominal
definition.
What are the different aspects (dimensions) of the construct?
Dimensions may be further subdivided in sub-dimensions.
Work on this until you have something measurable >
operational definition – a statement that maps one or more
empirical measure onto one or more theoretical constructs.
Example:
1. Nominal definition: ‘Social capital consists of networks of social
relations which are characterized by norms of trust and reciprocity.
Combined, it is these elements which are argued to sustain civil
society, and which enable people to act for mutual benefit’ (Stone,
2001).
2. Aspects/ (sub)dimensions:
Structure of social relationships > networks (type, size and capacity).
Quality of social relationships > norms (trust, reciprocity).
, Established methods of scale construction
1. Construct method (deductive)
o Deductive in nature
Based on a conceptual and
theoretical framework
Guided by testing hypotheses.
Items are derived from a
theoretical definition of the
construct.
o Nomological network - a theoretical
network of associations of the construct
with other variables derived from the
construct theory.
o Item production stage: operational
definition (of all variables).
o Construct validity: convergent and
divergent validity of the items important
for scale construction after a first administration of an item set.
o Aim: a homogenous set of items covering all aspects of the
construct.
How to evaluate? Empirically: MTMM matrix, CFA, reliability analysis.
2. Facet design method.
o Deductive in nature: items are derived from a mapping sentence.
o Facets and facet elements instead of nomological network.
o Content validity: evaluates how well an instrument covers all
relevant parts (the so-called facets) of the construct it aims to
measure dimensionality analysis.
1. Identifying the behavioral features or processes that are
essential to the construct.
2. Essential aspects > facets are specified.
Facets related to the content of the item: situational facets,
behavioral facets.
Facet related to the answer alternatives of the item.
3. For each facet, its elements are defined (called > structs).
4. The facets are completely cross classified (and each combination
of unique structs is called a structuple).
Facets should be independent and mutually exclusive!
Example: dental anxiety questionnaire > structuple (a1, b3, c1).
