Quantitative research methods summary
Week 1 – data
Measurement levels of data
Nominal – categorical, no order, categories differentiate only in name (peter, henk, jos)
Ordinal – categorical, categories differentiate in name and rank order (mavo, havo, vwo)
Interval – metric, distance between categories are equal (temperature: -5, 0, 5, 10 degrees)
Ratio – metric, distance between categories are equal WITH zero point (age: 0, 1, 2, 3)
Stephen Toulmin’s model of argumentation: claim based on a ground, which is based on
warrant. Claim: choice/decision/opinion, ground: information (statistical output,
measurement levels, results from previous research), warrant: rules and principles.
1) What is your decision/conclusion? (claim)
- Confirm or reject your hypothesis H0 and H1.
2) What data substantiate your conclusion? (ground)
- Ground is the data for your conclusion. Data can be several measurement levels.
3) Why is your conclusion correct based on that data? (warrant)
- General statement, based on rules or principles, why your conclusion is correct
based on your data.
Warrant
Ground Claim
Is there a correlation between two variables?
- In case of metric variables (interval or ratio), you use Pearson’s Correlation test
- Nul hypothesis (H0) means there is no correlation between the variables
- The alternative hypothesis (H1) means there is a correlation between the variables
When is the correlation significant? You reject H0 to accept H1 when the probability of the
test is smaller than the significance criteria you have used (usually, criteria/alpha is 0.05).
Exception of this alpha is in testing representativeness!
H0: no correlation between variables
H1: correlation between variables
Accept H0 when P > 0.05
Accept H1 when P < 0.05
Quality of research: three aspects
- Validity: the extent to which the characteristics measured are the actual
characteristics of the objects involved in the research (systematic or random error).
o Internal validity: degree to which you measure what you want to measure.
, o External validity: extent to which your results are generalizable to the
population of the sample.
- Reliability: the extent to which the measurements of the characteristics produce
equal results when the study would be repeated under the same conditions.
- Utility: the extent to which the results fit with the problem of the client, or the
extent to which the results actually contribute to solve the problem (usability).
Empirical cycle as a framework: the literature study and research problem are interrelated,
it develops a conceptual model, leading to observation/data collection, data analysis and
conclusion.
Quantitative data
Research strategies: a plan of the most important steps of your research process
- Survey; often used when looking for many variables in many respondents >
population census (make generalization about population), longitudinal
(developments over time) and ad hoc (for specific cases). Descriptive and exploratory
questions.
- Experiment; often used for causal relationships, explain effects (experiment groups)
Data collection: within each strategy, you can make use of different data collection methods
such as questionnaires, observations and content analysis. Questionnaires and observations
can be used in surveys and experiments, but in experiments not likely to do content analysis
(look at all kinds of sources like media, written sources, and find typical behavior).
Data:
- Primary data: collected for the current research, not collected before
- Secondary data: collected for another purpose, before your research, already
existing data.
o Advantages of secondary data: easy to access, cheap, available, expand
insight in primary data.
o Disadvantages of secondary data: not always valid or reliable, limited
documentation, measurements or definitions do not match your research,
outdated.
Questionnaire
Question types: open ended/closed questions, single/multiple response (tick one or more
answers), dichotomous questions (yes/no), scale items (e.g. Likert, agree/disagree).
Phrasing questions: use common language, use unambiguous words, avoid implicit
assumptions, avoid generalizations and estimations, use positively and negatively phrased
questions in Likert to prevent automatic answering mode).
Sampling: probability sampling (can be representative for population), non-probability
sampling (can not be representative for population).
Ethical issues: research ethics more and more important, so get approval from an ethical
committee, and consider: follow informed consent rules (explain the research and ask if they
want to participate), respect confidentiality and privacy (anonymously, data not public),
subject to change (how to address gender).
,Sampling explained: it is not possible to collect data from everyone, so you use only a part
of the population for data.
- Population: all people
- Operational population: part of the population you use in your research
- Sample frame: specific frame from where you draw your sample, category
- Initial sample: the sample group you want to use for your research
- Final sample: initial sample minus non-response
Non-probability samples: specifically selected sample, representativeness can’t be assumed
- Convenience sample: you choose the persons that are at your disposal
- Judgmental sample: you choose only specific groups and exclude others (age)
- Quota sample: you choose people based on quota, e.g. 50% male and 50% female
- Snowball sample: when it’s difficult to find people, search for one person and ask if
they now more people
Probability samples: samples based on faith, selected by chance
- Random sample: give total population a number and randomly draw lots
- Systematic sample with random start: determine a start number and continue to
select in a systematic way (e.g. 3-13-23-33-43)
- Stratified sample: draw a sample within specific groups of your research to make
sure both groups are represented
- Cluster sample: select a limited number and involve everyone from that number
Testing representativeness
= degree to which a sample adequately reflects the population for relevant characteristics
- When you test representativeness for categorical variables: use Chi square test
- When you test representativeness for metric variables: use T-test of one sample
How to use T-test for one sample: you compare the mean of the sample with the
expected/published variable in data for the population (CBS).
How to use Chi square test: compare the statistics on sample and expected frequencies.
1. Write down observed frequency FO.
2. Calculate expected frequency FE from external data. Uniform distribution means
frequency is equal for all subcategories, so you divide the total by the amount of
subcategories. 80 people divided over 3 supermarkets means 26,7 people per
supermarket.
Calculate Chi square = sum of ((observed value – experienced value)^2 / expected value)
So, calculate (FO-FE^2 / FE) for all categories and add them up.
How to know whether the score is significant and the sample is representative:
H0: representative – distribution in sample = distribution in population
H1: not representative – distribution in sample ≠ distribution in population
Degrees of freedom = number of categories - 1
, Outcome Chi2 (Degrees of Freedom, Number of cases = ) = x, p = …
X2(df, N= x) = answer; P = answer
Df (degree of freedom) = number of categories – 1
N = number of cases
P = probability value
Statistical decision is that alpha = 0.30, and when p > 0.30, H0 accepted and H1 rejected
Statistical decision is that alpha = 0.30, and when p < 0.30, H0 rejected and H1 accepted
So when p > 0.30, sample is representative, and when p < 0.30, sample is not representative
Is the p smaller than alpha (0.30) > reject H0. However, is p bigger than alpha (0.30), accept
H0 meaning that the sample is representative!
Procedure in SPSS:
Menu Analyze > Non-parametric tests > Legacy Dialogs > Chi-Square
Menu Analyze > Non-parametric tests > One Sample > etc.
SPSS output gives you observed value, expected value, Chi2 value, degrees of freedom and
probability!
Why testing at an alpha of 0.30 and not 0.05 when testing representativeness?
In a normal test, you don’t want to reject H0 too quickly, so you keep the alpha low.
However, in a representativity test, you don’t want to accept H0 too quickly, because that
would mean you assume representativeness when that’s not the case. So to avoid such a
mistake, you increase the alpha to 0.30, so it is more difficult to accept H0.
Reality
H0 is true H0 is not true
Statistical decision Accept H0 Correctly accepted Type II error,
incorrectly accepted
H0 and
representativeness.
Chance of this is B.
Accept H1 Type I error, Correctly accepted
incorrectly rejected
H0 and non-
representativeness.
Chance of this is
alpha.
Explanation: In the representativeness test, it is very important to not accept H0 wrongly.
This would be a type II error. It is more important to avoid type II error than to avoid type I
error. We don’t know ‘B’/power, but we do know that type I and type II errors are
interrelated: when alpha increases, b decreases. So if you want to decrease b, the chance of
a type II error, you have to increase alpha. With an alpha of 0,30 instead of 0,05, we
Week 1 – data
Measurement levels of data
Nominal – categorical, no order, categories differentiate only in name (peter, henk, jos)
Ordinal – categorical, categories differentiate in name and rank order (mavo, havo, vwo)
Interval – metric, distance between categories are equal (temperature: -5, 0, 5, 10 degrees)
Ratio – metric, distance between categories are equal WITH zero point (age: 0, 1, 2, 3)
Stephen Toulmin’s model of argumentation: claim based on a ground, which is based on
warrant. Claim: choice/decision/opinion, ground: information (statistical output,
measurement levels, results from previous research), warrant: rules and principles.
1) What is your decision/conclusion? (claim)
- Confirm or reject your hypothesis H0 and H1.
2) What data substantiate your conclusion? (ground)
- Ground is the data for your conclusion. Data can be several measurement levels.
3) Why is your conclusion correct based on that data? (warrant)
- General statement, based on rules or principles, why your conclusion is correct
based on your data.
Warrant
Ground Claim
Is there a correlation between two variables?
- In case of metric variables (interval or ratio), you use Pearson’s Correlation test
- Nul hypothesis (H0) means there is no correlation between the variables
- The alternative hypothesis (H1) means there is a correlation between the variables
When is the correlation significant? You reject H0 to accept H1 when the probability of the
test is smaller than the significance criteria you have used (usually, criteria/alpha is 0.05).
Exception of this alpha is in testing representativeness!
H0: no correlation between variables
H1: correlation between variables
Accept H0 when P > 0.05
Accept H1 when P < 0.05
Quality of research: three aspects
- Validity: the extent to which the characteristics measured are the actual
characteristics of the objects involved in the research (systematic or random error).
o Internal validity: degree to which you measure what you want to measure.
, o External validity: extent to which your results are generalizable to the
population of the sample.
- Reliability: the extent to which the measurements of the characteristics produce
equal results when the study would be repeated under the same conditions.
- Utility: the extent to which the results fit with the problem of the client, or the
extent to which the results actually contribute to solve the problem (usability).
Empirical cycle as a framework: the literature study and research problem are interrelated,
it develops a conceptual model, leading to observation/data collection, data analysis and
conclusion.
Quantitative data
Research strategies: a plan of the most important steps of your research process
- Survey; often used when looking for many variables in many respondents >
population census (make generalization about population), longitudinal
(developments over time) and ad hoc (for specific cases). Descriptive and exploratory
questions.
- Experiment; often used for causal relationships, explain effects (experiment groups)
Data collection: within each strategy, you can make use of different data collection methods
such as questionnaires, observations and content analysis. Questionnaires and observations
can be used in surveys and experiments, but in experiments not likely to do content analysis
(look at all kinds of sources like media, written sources, and find typical behavior).
Data:
- Primary data: collected for the current research, not collected before
- Secondary data: collected for another purpose, before your research, already
existing data.
o Advantages of secondary data: easy to access, cheap, available, expand
insight in primary data.
o Disadvantages of secondary data: not always valid or reliable, limited
documentation, measurements or definitions do not match your research,
outdated.
Questionnaire
Question types: open ended/closed questions, single/multiple response (tick one or more
answers), dichotomous questions (yes/no), scale items (e.g. Likert, agree/disagree).
Phrasing questions: use common language, use unambiguous words, avoid implicit
assumptions, avoid generalizations and estimations, use positively and negatively phrased
questions in Likert to prevent automatic answering mode).
Sampling: probability sampling (can be representative for population), non-probability
sampling (can not be representative for population).
Ethical issues: research ethics more and more important, so get approval from an ethical
committee, and consider: follow informed consent rules (explain the research and ask if they
want to participate), respect confidentiality and privacy (anonymously, data not public),
subject to change (how to address gender).
,Sampling explained: it is not possible to collect data from everyone, so you use only a part
of the population for data.
- Population: all people
- Operational population: part of the population you use in your research
- Sample frame: specific frame from where you draw your sample, category
- Initial sample: the sample group you want to use for your research
- Final sample: initial sample minus non-response
Non-probability samples: specifically selected sample, representativeness can’t be assumed
- Convenience sample: you choose the persons that are at your disposal
- Judgmental sample: you choose only specific groups and exclude others (age)
- Quota sample: you choose people based on quota, e.g. 50% male and 50% female
- Snowball sample: when it’s difficult to find people, search for one person and ask if
they now more people
Probability samples: samples based on faith, selected by chance
- Random sample: give total population a number and randomly draw lots
- Systematic sample with random start: determine a start number and continue to
select in a systematic way (e.g. 3-13-23-33-43)
- Stratified sample: draw a sample within specific groups of your research to make
sure both groups are represented
- Cluster sample: select a limited number and involve everyone from that number
Testing representativeness
= degree to which a sample adequately reflects the population for relevant characteristics
- When you test representativeness for categorical variables: use Chi square test
- When you test representativeness for metric variables: use T-test of one sample
How to use T-test for one sample: you compare the mean of the sample with the
expected/published variable in data for the population (CBS).
How to use Chi square test: compare the statistics on sample and expected frequencies.
1. Write down observed frequency FO.
2. Calculate expected frequency FE from external data. Uniform distribution means
frequency is equal for all subcategories, so you divide the total by the amount of
subcategories. 80 people divided over 3 supermarkets means 26,7 people per
supermarket.
Calculate Chi square = sum of ((observed value – experienced value)^2 / expected value)
So, calculate (FO-FE^2 / FE) for all categories and add them up.
How to know whether the score is significant and the sample is representative:
H0: representative – distribution in sample = distribution in population
H1: not representative – distribution in sample ≠ distribution in population
Degrees of freedom = number of categories - 1
, Outcome Chi2 (Degrees of Freedom, Number of cases = ) = x, p = …
X2(df, N= x) = answer; P = answer
Df (degree of freedom) = number of categories – 1
N = number of cases
P = probability value
Statistical decision is that alpha = 0.30, and when p > 0.30, H0 accepted and H1 rejected
Statistical decision is that alpha = 0.30, and when p < 0.30, H0 rejected and H1 accepted
So when p > 0.30, sample is representative, and when p < 0.30, sample is not representative
Is the p smaller than alpha (0.30) > reject H0. However, is p bigger than alpha (0.30), accept
H0 meaning that the sample is representative!
Procedure in SPSS:
Menu Analyze > Non-parametric tests > Legacy Dialogs > Chi-Square
Menu Analyze > Non-parametric tests > One Sample > etc.
SPSS output gives you observed value, expected value, Chi2 value, degrees of freedom and
probability!
Why testing at an alpha of 0.30 and not 0.05 when testing representativeness?
In a normal test, you don’t want to reject H0 too quickly, so you keep the alpha low.
However, in a representativity test, you don’t want to accept H0 too quickly, because that
would mean you assume representativeness when that’s not the case. So to avoid such a
mistake, you increase the alpha to 0.30, so it is more difficult to accept H0.
Reality
H0 is true H0 is not true
Statistical decision Accept H0 Correctly accepted Type II error,
incorrectly accepted
H0 and
representativeness.
Chance of this is B.
Accept H1 Type I error, Correctly accepted
incorrectly rejected
H0 and non-
representativeness.
Chance of this is
alpha.
Explanation: In the representativeness test, it is very important to not accept H0 wrongly.
This would be a type II error. It is more important to avoid type II error than to avoid type I
error. We don’t know ‘B’/power, but we do know that type I and type II errors are
interrelated: when alpha increases, b decreases. So if you want to decrease b, the chance of
a type II error, you have to increase alpha. With an alpha of 0,30 instead of 0,05, we
