Necessities- Survey Research
Survey = A systematic method to gather information about the properties of a sample in order to make
quantitative inferences about the properties of the population to which these belong, predominantly by
asking questions to the entities.
The goal of survey methodology (and of the survey validity model): Maximizing the validity of the
inferences from surveys by minimizing the sources of error in survey research.
Three functions of survey research:
1. Knowledge
2. Influence
3. Entertainment
Three goals of survey methodology:
1. Measurement
a. Properties - Characteristics and relationships in the population
b. Conceptual model - Constructs (latent variable) and their relationships
c. Instrument – Form in which overt variable is assessed
d. Response - Response provided
e. Edited response
2. Representation
a. Population - Set of units under study
b. Sampling frame - Set of eligible units to survey
c. Sample - Set of units to survey
d. Respondents - Set of successfully surveyed units
e. Adjusted Respondents - Final set of respondents for analysis
3. Analysis
Is validity only present or important in the measurement part? NO!! In all 3 parts (measurement,
representation, analysis & reporting)
Mapping (M) Fitting one’s true internal response(s) on the response options, and/or matching the
response options to the true internal response.
Responding (R) Providing the external response – one error: strategically editing one’s true response.
The 10 errors of surveying
I) Conceptualization error – constructs do not map on properties
II) Operationalization error – systematic departure from the constructs – an invalid measure / a bad
scale
III) Coverage error – sampling frame does not capture the population
IV) Sampling error – deliberate error because of cost or feasibility - You draw a lot of different
samples and hope it together forms a representative sample of the total population
V) Measurement error – systematic departure of measurement from true value of the constructs
and their relationship due to measurement process itself. – consumers interpret question wrong
Necessities Survey Research – Eline van de Ven
,VI) Processing error – systematic departure of edited responses from true value of constructs due to
editing
VII) Nonresponse error – when non-respondents systematically differ from respondents
VIII) Adjustment error – biased attempts to correct for processing, coverage, sampling, and
nonresponse errors
IX) Analysis and reporting error – systematic mistakes in analysis and reporting that reduce the
validity of findings.
The strengths of the survey validity model: IDTC
1. Integrated model of validity
2. Decomposes the total error into specific sources
3. Tools to prevent errors and cope with these
4. Conceptual foundation of survey methodology
Necessities Survey Research – Eline van de Ven
, Validity = accuracy – low bias Does it measure what it’s supposed to?
Reliability = replicability – low variance Is it stable?
What types of validity exist?
– Maximum accuracy (maximize accuracy of your measures)
– Minimal bias (bias = difference between what is true and what you think is)
What types of reliability exist?
– Maximum replicability (test it again, is it the same?)
– Minimal variance
A conceptual model specifies: CMMCSS
1. Causation: Directed and undirected relationships
2. Mediation: Direct and indirect effects
3. Moderation: Unconditional and conditional effects
4. Controls: Confounders / covariates
5. Status: Attributes and constructs
6. Sign: Valence of effects, + or -
Endogeneity Bias: An endogenous variable is false assumed to be exogenous. Thus, X is caused by another
variable. Three sources:
1. Reverse or simultaneous causality : Simultaneity Bias
2. Omitted confounders : Omitted Variable Bias
3. Measurement error : Attenuation Bias
Solutions:
1. Reverse causality and simultaneity
a. Explain that A can affect B but B can affect A – Develop strong theory about the causality
b. Identify natural experiments – lotteries, unexpected unemployment
c. Longitudinal studies; time helps to identify causality
2. Omitted variables
a. Add potential confounding variables
3. Measurement error
Necessities Survey Research – Eline van de Ven
Survey = A systematic method to gather information about the properties of a sample in order to make
quantitative inferences about the properties of the population to which these belong, predominantly by
asking questions to the entities.
The goal of survey methodology (and of the survey validity model): Maximizing the validity of the
inferences from surveys by minimizing the sources of error in survey research.
Three functions of survey research:
1. Knowledge
2. Influence
3. Entertainment
Three goals of survey methodology:
1. Measurement
a. Properties - Characteristics and relationships in the population
b. Conceptual model - Constructs (latent variable) and their relationships
c. Instrument – Form in which overt variable is assessed
d. Response - Response provided
e. Edited response
2. Representation
a. Population - Set of units under study
b. Sampling frame - Set of eligible units to survey
c. Sample - Set of units to survey
d. Respondents - Set of successfully surveyed units
e. Adjusted Respondents - Final set of respondents for analysis
3. Analysis
Is validity only present or important in the measurement part? NO!! In all 3 parts (measurement,
representation, analysis & reporting)
Mapping (M) Fitting one’s true internal response(s) on the response options, and/or matching the
response options to the true internal response.
Responding (R) Providing the external response – one error: strategically editing one’s true response.
The 10 errors of surveying
I) Conceptualization error – constructs do not map on properties
II) Operationalization error – systematic departure from the constructs – an invalid measure / a bad
scale
III) Coverage error – sampling frame does not capture the population
IV) Sampling error – deliberate error because of cost or feasibility - You draw a lot of different
samples and hope it together forms a representative sample of the total population
V) Measurement error – systematic departure of measurement from true value of the constructs
and their relationship due to measurement process itself. – consumers interpret question wrong
Necessities Survey Research – Eline van de Ven
,VI) Processing error – systematic departure of edited responses from true value of constructs due to
editing
VII) Nonresponse error – when non-respondents systematically differ from respondents
VIII) Adjustment error – biased attempts to correct for processing, coverage, sampling, and
nonresponse errors
IX) Analysis and reporting error – systematic mistakes in analysis and reporting that reduce the
validity of findings.
The strengths of the survey validity model: IDTC
1. Integrated model of validity
2. Decomposes the total error into specific sources
3. Tools to prevent errors and cope with these
4. Conceptual foundation of survey methodology
Necessities Survey Research – Eline van de Ven
, Validity = accuracy – low bias Does it measure what it’s supposed to?
Reliability = replicability – low variance Is it stable?
What types of validity exist?
– Maximum accuracy (maximize accuracy of your measures)
– Minimal bias (bias = difference between what is true and what you think is)
What types of reliability exist?
– Maximum replicability (test it again, is it the same?)
– Minimal variance
A conceptual model specifies: CMMCSS
1. Causation: Directed and undirected relationships
2. Mediation: Direct and indirect effects
3. Moderation: Unconditional and conditional effects
4. Controls: Confounders / covariates
5. Status: Attributes and constructs
6. Sign: Valence of effects, + or -
Endogeneity Bias: An endogenous variable is false assumed to be exogenous. Thus, X is caused by another
variable. Three sources:
1. Reverse or simultaneous causality : Simultaneity Bias
2. Omitted confounders : Omitted Variable Bias
3. Measurement error : Attenuation Bias
Solutions:
1. Reverse causality and simultaneity
a. Explain that A can affect B but B can affect A – Develop strong theory about the causality
b. Identify natural experiments – lotteries, unexpected unemployment
c. Longitudinal studies; time helps to identify causality
2. Omitted variables
a. Add potential confounding variables
3. Measurement error
Necessities Survey Research – Eline van de Ven
