Construction and Analysis of Questionnaires
Lecture 1
Questionnaire is a written set of questions that are given to people in order to collect facts or opinions
about something (typical performance). One of the main data collection tools.
Interactional mode: with interviewer or self-administered
Technological mode: telephone, computer, internet, paper-and-pencil
- PAPI: Paper and Pencil Interviewing - CATI: Computer Assisted Telephone Interviewing
- CAPI: Computer Assisted Personal Interviewing - TAPI: Tablet Assisted Personal Interviewing
- WAPI: Web Assisted Personal Interviewing - TASI: Tablet Assisted Self Interviewing
- CASI: Computer Assisted Self Interviewing - SAPI: Smartphone Assisted Personal Interviewing
- CAWI: Computer Assisted Web Interviewing
+ Fast way of obtaining data, cheap, can be used for large samples
- Low response rates (especially for mail&internet), answer to question may deviate from actual facts
Survey is a technique/research method for collecting information characterized by (1) structured or
systematic set of data (variable x case data grid) (2) seeking an understanding of what may cause a
phenomenon. Information may be collected using a questionnaire but may also be obtained using
an interview or observation for example. Survey ≠ Questionnaire
Test is mainly understood as an achievement test. Often in the form of a series of questions where
the answers are qualified in terms of ‘right and wrong’. Asks about maximal performance and
questionnaires ask about typical performance.
Lecture 2 – Questionnaire construction: developing indicators for concepts
Social research is often based on relations between concepts.
Concepts are mental representations and are typically based on experience; abstract summaries of
a whole set of characteristics, behaviors, attitudes we see as having something in common. a
problem with this is that most concepts cannot be measured in a direct way, but are latent solution
for this is indirect measurement using proxies or so-called indicators: operationalization of the
(abstract) concept into a measurable variable.
Heuristic approach: descending the ladder of abstraction. Develop our own nominal
definition and distinguish different aspects/dimensions (sub-dimensions). Until we get to the
point where we can develop indicators (=measurable) operational definition
Index is a summary measure of observable characteristics/indicators. Often concepts are not
observable but latent (= construct). These are measured using scales = summary measure of
observable characteristics / indicators for the construct / instrument to measure a construct or
dimension (sub-construct).
, Scale Construction
Construct Method Deductive, starts from a theoretical framework. Specifications of the
nomological network: the construct is studied in detail, a precise definition
is given. Homogenous set of items produced that cover all aspects. Lot of
attention for convergent and divergent validity.
Facet Method Deductive, conceptual analysis (4 steps) (1) Analysis of the behaviors
essential for the construct. (2) Essential aspects are specified, called facets.
(3) For each facet a number of facet elements is defined structs. (4) the
facets are completely cross-classified each resulting element is called a
structuple mappings sentence (karteringszin)
Rational Method Deductive, expert knowledge is used to define the construct and to assess
the items. Often used and high face validity = extent to which one can tell
what the item is measuring.
Internal Method Inductive: there is no analysis of the content of a construct, there is no
theory used. A very large set of items is created using any method and
combinations. Construction of scale and selection of items using methods
for reliability and validity. Also known as internal consistency method.
Prototypical Inductive, no content-analysis of the construct, no theory involved. Main
Method goal is to represent the most typical elements of the construct. Item
production: act-nomination by respondents (experts). Non-experts rate the
extent to which an item fits the construct. Act-frequency approach:
summary of trend of someone’s behavior, feelings, thoughts.
External Method Inductive. Construction of items is largely the same as internal method yet
there is a present idea on what the items have to be related to (external
criterion). Selection of items: optimize relationship with an external criterion.
This is maximizing the criterion oriented validity.
Lecture 3 – Questionnaire Construction
Item is the statement / question including the response alternatives. Rules for formulating items:
1. Use clear and comprehensible (simple) wording. Avoid vague terms, difficult expressions and
jargon. The item should not require reading skills beyond that of a 12 year old boy)
2. Use short statements/questions
3. Do not use double-barreled questions, just ask one question per item
4. Write questions/statements that are precise, this means to add context, time, numbers, dates
5. Make sure the respondent has the necessary knowledge/meets the necessary conditions
6. Avoid negative questions ‘not’, ‘never’, ‘nothing, ‘nobody’ indicative & contra–indicative
7. Use about as many indicative as contra-indicative items, to avoid acquiescence bias this is
the tendency to agree with every statement
8. Make sure the words have the same meaning to everyone
, 9. If a statement/question has a condition and a behavior part, start with the condition part
10. Avoid leading questions, structure/wording of these question pushes people to give an answer
that they would not have given if the question had been askes in a more neutral way
Rules for formulating answer alternatives:
1. The answer alternatives should be exhaustive
2. The answer alternatives should be mutually exclusive, means that person can provide one and
only one answer to the question.
3. Avoid open questions as much as possible, they are difficult to analyze
4. Put the alternatives in a logical order
5. If possible avoid ‘don’t know’, ‘not applicable’, ‘no answer’. These kind of answers stimulate to
be lazy (‘satisficing’) so force the subject to give his opinion/look for the answer. Yet what if he
truly has no opinion. This creates false and unreliable answers
6. Only one answer instead of allowing for multiple answers
7. Use, as much as possible, the same response alternatives. Easier for respondents & less error
8. Preferable format: bipolar, symmetric, 5 or 7 response alternatives:
Bipolar: endpoint have opposite meanings
Unipolar: express frequency/quantity
Symmetry: equal number of positive and negative question, equally spaced from ‘neutral’
Balance: distance between alternatives is the same mean over items is meaningful
Likert questionnaire item, respondents specify their level of agreement or disagreement on a
symmetric agree-disagree scale for a series of statements. A Likert Scale can be created as a
simple sum of questionnaire responses over the full range of the scale. In doing so, Likert scaling
assumes distances between each item are equal (balance).
Many of the rules and guidelines in questionnaire construction and item writing are motivated by
the following behavior of the respondents:
- Social Desirability = the tendency to answer questionnaire items so that people look good in
their own eyes and in the eyes of others. Social desirable behavior is over-reported and
undesirable is under-reported. Occurs more often in face-to-face interviewing.
- Acquiescence (instemmen) = agreement with statements as presented, greatest among
children and respondents with low education. Balance the number of indicative and contra-
indicative items of the agree/disagree format.
- Satisficing (satisfy + suffice) = limitations in cognitive ability and/or cognitive effort invested
make that people give an answer that is sufficient and do not attempt to go for the ‘optimal’
answer. Make items easy and motivate people.
Lecture 1
Questionnaire is a written set of questions that are given to people in order to collect facts or opinions
about something (typical performance). One of the main data collection tools.
Interactional mode: with interviewer or self-administered
Technological mode: telephone, computer, internet, paper-and-pencil
- PAPI: Paper and Pencil Interviewing - CATI: Computer Assisted Telephone Interviewing
- CAPI: Computer Assisted Personal Interviewing - TAPI: Tablet Assisted Personal Interviewing
- WAPI: Web Assisted Personal Interviewing - TASI: Tablet Assisted Self Interviewing
- CASI: Computer Assisted Self Interviewing - SAPI: Smartphone Assisted Personal Interviewing
- CAWI: Computer Assisted Web Interviewing
+ Fast way of obtaining data, cheap, can be used for large samples
- Low response rates (especially for mail&internet), answer to question may deviate from actual facts
Survey is a technique/research method for collecting information characterized by (1) structured or
systematic set of data (variable x case data grid) (2) seeking an understanding of what may cause a
phenomenon. Information may be collected using a questionnaire but may also be obtained using
an interview or observation for example. Survey ≠ Questionnaire
Test is mainly understood as an achievement test. Often in the form of a series of questions where
the answers are qualified in terms of ‘right and wrong’. Asks about maximal performance and
questionnaires ask about typical performance.
Lecture 2 – Questionnaire construction: developing indicators for concepts
Social research is often based on relations between concepts.
Concepts are mental representations and are typically based on experience; abstract summaries of
a whole set of characteristics, behaviors, attitudes we see as having something in common. a
problem with this is that most concepts cannot be measured in a direct way, but are latent solution
for this is indirect measurement using proxies or so-called indicators: operationalization of the
(abstract) concept into a measurable variable.
Heuristic approach: descending the ladder of abstraction. Develop our own nominal
definition and distinguish different aspects/dimensions (sub-dimensions). Until we get to the
point where we can develop indicators (=measurable) operational definition
Index is a summary measure of observable characteristics/indicators. Often concepts are not
observable but latent (= construct). These are measured using scales = summary measure of
observable characteristics / indicators for the construct / instrument to measure a construct or
dimension (sub-construct).
, Scale Construction
Construct Method Deductive, starts from a theoretical framework. Specifications of the
nomological network: the construct is studied in detail, a precise definition
is given. Homogenous set of items produced that cover all aspects. Lot of
attention for convergent and divergent validity.
Facet Method Deductive, conceptual analysis (4 steps) (1) Analysis of the behaviors
essential for the construct. (2) Essential aspects are specified, called facets.
(3) For each facet a number of facet elements is defined structs. (4) the
facets are completely cross-classified each resulting element is called a
structuple mappings sentence (karteringszin)
Rational Method Deductive, expert knowledge is used to define the construct and to assess
the items. Often used and high face validity = extent to which one can tell
what the item is measuring.
Internal Method Inductive: there is no analysis of the content of a construct, there is no
theory used. A very large set of items is created using any method and
combinations. Construction of scale and selection of items using methods
for reliability and validity. Also known as internal consistency method.
Prototypical Inductive, no content-analysis of the construct, no theory involved. Main
Method goal is to represent the most typical elements of the construct. Item
production: act-nomination by respondents (experts). Non-experts rate the
extent to which an item fits the construct. Act-frequency approach:
summary of trend of someone’s behavior, feelings, thoughts.
External Method Inductive. Construction of items is largely the same as internal method yet
there is a present idea on what the items have to be related to (external
criterion). Selection of items: optimize relationship with an external criterion.
This is maximizing the criterion oriented validity.
Lecture 3 – Questionnaire Construction
Item is the statement / question including the response alternatives. Rules for formulating items:
1. Use clear and comprehensible (simple) wording. Avoid vague terms, difficult expressions and
jargon. The item should not require reading skills beyond that of a 12 year old boy)
2. Use short statements/questions
3. Do not use double-barreled questions, just ask one question per item
4. Write questions/statements that are precise, this means to add context, time, numbers, dates
5. Make sure the respondent has the necessary knowledge/meets the necessary conditions
6. Avoid negative questions ‘not’, ‘never’, ‘nothing, ‘nobody’ indicative & contra–indicative
7. Use about as many indicative as contra-indicative items, to avoid acquiescence bias this is
the tendency to agree with every statement
8. Make sure the words have the same meaning to everyone
, 9. If a statement/question has a condition and a behavior part, start with the condition part
10. Avoid leading questions, structure/wording of these question pushes people to give an answer
that they would not have given if the question had been askes in a more neutral way
Rules for formulating answer alternatives:
1. The answer alternatives should be exhaustive
2. The answer alternatives should be mutually exclusive, means that person can provide one and
only one answer to the question.
3. Avoid open questions as much as possible, they are difficult to analyze
4. Put the alternatives in a logical order
5. If possible avoid ‘don’t know’, ‘not applicable’, ‘no answer’. These kind of answers stimulate to
be lazy (‘satisficing’) so force the subject to give his opinion/look for the answer. Yet what if he
truly has no opinion. This creates false and unreliable answers
6. Only one answer instead of allowing for multiple answers
7. Use, as much as possible, the same response alternatives. Easier for respondents & less error
8. Preferable format: bipolar, symmetric, 5 or 7 response alternatives:
Bipolar: endpoint have opposite meanings
Unipolar: express frequency/quantity
Symmetry: equal number of positive and negative question, equally spaced from ‘neutral’
Balance: distance between alternatives is the same mean over items is meaningful
Likert questionnaire item, respondents specify their level of agreement or disagreement on a
symmetric agree-disagree scale for a series of statements. A Likert Scale can be created as a
simple sum of questionnaire responses over the full range of the scale. In doing so, Likert scaling
assumes distances between each item are equal (balance).
Many of the rules and guidelines in questionnaire construction and item writing are motivated by
the following behavior of the respondents:
- Social Desirability = the tendency to answer questionnaire items so that people look good in
their own eyes and in the eyes of others. Social desirable behavior is over-reported and
undesirable is under-reported. Occurs more often in face-to-face interviewing.
- Acquiescence (instemmen) = agreement with statements as presented, greatest among
children and respondents with low education. Balance the number of indicative and contra-
indicative items of the agree/disagree format.
- Satisficing (satisfy + suffice) = limitations in cognitive ability and/or cognitive effort invested
make that people give an answer that is sufficient and do not attempt to go for the ‘optimal’
answer. Make items easy and motivate people.
