Construction and Analysis of Questionnaires Summary
Lecture 1
Two types of performances measured:
1.Maximum performance: what a person can do (Achievement test)
Examples: aptitude tests, achievement tests, intelligent tests
Focus on: obtaining highest scores (determined by abilities)
2.Typical Performance: What a person will do (Questionnaire)
Examples: Personality questionnaires, behavioural questionnaires
Focus on obtaining representative responses (no right or wrong answers)
Item: statement (or question) and the response alternatives
One measurement instrument: tries to isolate ONE characteristic and tries to quantify to what extent
someone ‘has that’.
Gives an incomplete description of reality (on purpose)
Examples: IQ score test, Intelligence test
Latent variable: not directly observable, use questionnaires (example: depression)
Validity: indicates whether a study's findings are trustworthy!
→ Is what we measure truly what we intend to measure?
Construct validity: how well a test measures the concept it was designed to evaluate
Convergent validity: measures whether constructs that theoretically should be related to
each other are, in fact, related to each other
Divergent validity: shows you whether two tests that should not be highly related to each
other are, indeed, unrelated.
Content validity: evaluates how well an instrument covers all relevant parts
Reliability: is about the precision and repeatability of the measurement!
→ Will I always get a consistent result (can I replicate my result) if I measure something repeatedly
and under the same conditions?
→ Is there a lot of random measurement error?
Lecture 2: Making Constructs Measurable: Methods for
Questionnaire Design
Constructs: abstract summaries (mental representations) of a number of characteristics, behaviours,
and attitudes that share something in common.
abstract latent (not directly observable or measurable)
Examples: Justice, Beauty, Happiness, Health
,Making constructs measurable
You have two options:
1. Use an existing questionnaire
2. Create your own set of questionnaire items
Heuristic approach (‘Descending the ladder of abstraction’)
• Define what you mean by the given concept nominal definition!
• What are the different aspects (dimensions) of the construct?
• Work on this until you have something measurable operational definition: a statement that maps
one or more empirical measures onto one or more (!) theoretical constructs
Six Methods for Questionnaire Design:
1.Construct Method
-Deductive in nature:
• based on a conceptual and theoretical framework
• guided by testing hypotheses
Items are derived from a theoretical definition of the construct!
-Nomological network: a theoretical network of associations of the construct with other variables
derived from the construct theory
-Item production stage: operational definition (of all variables)
-Construct validity: convergent and divergent validity of the items important for scale construction
after a first administration of an item set
-Aim: A homogeneous set of items covering all aspects of the construct! How to evaluate it?
Empirically: MTMM matrix, CFA, reliability analysis
2.Facet Method
-Deductive in nature: Items are derived from a mapping sentence!
-Facets and facet elements instead of nomological network!
-Content validity: evaluates how well an instrument covers all relevant parts (the so-called facets) of
the construct it aims to measure dimensionality analysis!
STEP 1: Identifying the behavioral features or processes that are essential to the construct
STEP 2: Essential aspects FACETS are specified
• Facets related to the content of the item: situational facets, behavioural facets
• Facet related to the answer alternatives of the item
STEP 3: For each facet, its elements is defined (called STRUCTS)
STEP 4: The facets are completely cross-classified (and each combination of unique structs
is called a STRUCTUPLE)
, (A facet is for example a1)
3.Rational Method
-Intuitive in nature: Items are derived from informal criteria (working definition)
-The knowledge of experts and/or respondents plays a crucial role in defining the construct and
creating the items! → What kind of knowledge?
• Often used in clinical settings (diagnostic comparison, clinical cases, syndromes)
-Each item is assessed with respect to its face validity for measuring the construct.
• Example: Statements like ‘Anxious children are sensitive and need to be protected.’, etc.
4.Prototypical Method (Act frequency)
-Intuitive in nature: Items are created based on act (= behaviors) nomination!
-Informal knowledge and experience of respondents plays a key role in defining the construct and
creating the items! NO THEORY!
-Typical behaviors (acts) that exemplify a construct, some are more prototypical of the construct than
others! → process validity for difficult constructs
Think of persons with extreme positions on the construct to be operationalized, and write
down behaviors that exemplify this construct.
-Large sample of judges: rate the extent to which an item ‘fits’ the construct. The prototypical
method is systematic and extensive, using standardized evaluations. Main goal is to represent the
most (proto)typical acts of the construct.
Act nomination: Question to undergrads ‘Think of the three most dominant (fe)males you know. With
these individuals in mind, write down five acts or behaviors they have performed that reflect or
exemplify their dominance.’
• Based on act nomination, a list of 100 acts was derived.
• Scale construction based on prototypicality ratings: Judges rate on an x-point scale (e.g., 7-
point scale) how good the act represents the disposition
5.Internal Method (Factor Analytic)
Lecture 1
Two types of performances measured:
1.Maximum performance: what a person can do (Achievement test)
Examples: aptitude tests, achievement tests, intelligent tests
Focus on: obtaining highest scores (determined by abilities)
2.Typical Performance: What a person will do (Questionnaire)
Examples: Personality questionnaires, behavioural questionnaires
Focus on obtaining representative responses (no right or wrong answers)
Item: statement (or question) and the response alternatives
One measurement instrument: tries to isolate ONE characteristic and tries to quantify to what extent
someone ‘has that’.
Gives an incomplete description of reality (on purpose)
Examples: IQ score test, Intelligence test
Latent variable: not directly observable, use questionnaires (example: depression)
Validity: indicates whether a study's findings are trustworthy!
→ Is what we measure truly what we intend to measure?
Construct validity: how well a test measures the concept it was designed to evaluate
Convergent validity: measures whether constructs that theoretically should be related to
each other are, in fact, related to each other
Divergent validity: shows you whether two tests that should not be highly related to each
other are, indeed, unrelated.
Content validity: evaluates how well an instrument covers all relevant parts
Reliability: is about the precision and repeatability of the measurement!
→ Will I always get a consistent result (can I replicate my result) if I measure something repeatedly
and under the same conditions?
→ Is there a lot of random measurement error?
Lecture 2: Making Constructs Measurable: Methods for
Questionnaire Design
Constructs: abstract summaries (mental representations) of a number of characteristics, behaviours,
and attitudes that share something in common.
abstract latent (not directly observable or measurable)
Examples: Justice, Beauty, Happiness, Health
,Making constructs measurable
You have two options:
1. Use an existing questionnaire
2. Create your own set of questionnaire items
Heuristic approach (‘Descending the ladder of abstraction’)
• Define what you mean by the given concept nominal definition!
• What are the different aspects (dimensions) of the construct?
• Work on this until you have something measurable operational definition: a statement that maps
one or more empirical measures onto one or more (!) theoretical constructs
Six Methods for Questionnaire Design:
1.Construct Method
-Deductive in nature:
• based on a conceptual and theoretical framework
• guided by testing hypotheses
Items are derived from a theoretical definition of the construct!
-Nomological network: a theoretical network of associations of the construct with other variables
derived from the construct theory
-Item production stage: operational definition (of all variables)
-Construct validity: convergent and divergent validity of the items important for scale construction
after a first administration of an item set
-Aim: A homogeneous set of items covering all aspects of the construct! How to evaluate it?
Empirically: MTMM matrix, CFA, reliability analysis
2.Facet Method
-Deductive in nature: Items are derived from a mapping sentence!
-Facets and facet elements instead of nomological network!
-Content validity: evaluates how well an instrument covers all relevant parts (the so-called facets) of
the construct it aims to measure dimensionality analysis!
STEP 1: Identifying the behavioral features or processes that are essential to the construct
STEP 2: Essential aspects FACETS are specified
• Facets related to the content of the item: situational facets, behavioural facets
• Facet related to the answer alternatives of the item
STEP 3: For each facet, its elements is defined (called STRUCTS)
STEP 4: The facets are completely cross-classified (and each combination of unique structs
is called a STRUCTUPLE)
, (A facet is for example a1)
3.Rational Method
-Intuitive in nature: Items are derived from informal criteria (working definition)
-The knowledge of experts and/or respondents plays a crucial role in defining the construct and
creating the items! → What kind of knowledge?
• Often used in clinical settings (diagnostic comparison, clinical cases, syndromes)
-Each item is assessed with respect to its face validity for measuring the construct.
• Example: Statements like ‘Anxious children are sensitive and need to be protected.’, etc.
4.Prototypical Method (Act frequency)
-Intuitive in nature: Items are created based on act (= behaviors) nomination!
-Informal knowledge and experience of respondents plays a key role in defining the construct and
creating the items! NO THEORY!
-Typical behaviors (acts) that exemplify a construct, some are more prototypical of the construct than
others! → process validity for difficult constructs
Think of persons with extreme positions on the construct to be operationalized, and write
down behaviors that exemplify this construct.
-Large sample of judges: rate the extent to which an item ‘fits’ the construct. The prototypical
method is systematic and extensive, using standardized evaluations. Main goal is to represent the
most (proto)typical acts of the construct.
Act nomination: Question to undergrads ‘Think of the three most dominant (fe)males you know. With
these individuals in mind, write down five acts or behaviors they have performed that reflect or
exemplify their dominance.’
• Based on act nomination, a list of 100 acts was derived.
• Scale construction based on prototypicality ratings: Judges rate on an x-point scale (e.g., 7-
point scale) how good the act represents the disposition
5.Internal Method (Factor Analytic)
