Philosophy of Statistics 2021-
2022
Lecture 1: Problem of induction, Popper, and significance testing
Econometrics: Application of statistical techniques with the purpose of
making
inferences about economic phenomena.
Skepticism about econometrics
1. Unreliable (Keynes)? In physics, we expect mathematical regularity,
economics is about social phenomena: what happens when people
interact these do not follow mathematics/statistics relationships, so
they are the wrong tool.
Is it unreliable?
Reliability: Reliability describes whether the results of an analysis
are
expected to be the same if it is repeated under the
same
circumstances.
We do not know since we use observational data to construct and test
models, since we cannot replicate the economy.
Replication crisis: Significant outcomes of experiments could not be
replicated in
different settings. Findings were largely exaggerated.
2. Trivial? We can only discover things in the data we already know from
theoretical reasoning.
Is it trivial? No, if we want to quantify certain phenomena (for
answering certain policy questions) we need to look at the data, theory
only tells us if it will happen (or not). Take e.g. the Rheinhart-Rogoff
controversy, where a mistake in a research paper led to countries
cutting government cost in 2008, believing (incorrectly) this was best
for the economy. This had disastrous consequences for a lot of people.
3. Misleading? “Lies, damned lies and statistics” fitting statistics to
your view instead of checking your view by using statistics is not
objective.
Is it misleading?
,Objectivity: Something is said to be objective when it does not
depend on
the particular characteristics of individuals who are
involved.
There are many choices: data, models, statistical methods, …. and it
turns out that econometricians come to different answers with the
same case. There are objective better and worse choices, however.
Inductive inference and the problem of induction
We have two types of data:
Observational data: no interference of the researcher in general.
Experimental data: interference of the econometrician in general.
Inductive inference: Process of drawing a general conclusion on the
basis of
observations (e.g. all pigeons can fly).
The problem of induction is that there is no certainty in general
observations (watch out for confirmation bias), but inductive reasoning
can be reliable. We should not feel certain about the knowledge we have,
but we can be more or less confident about certain things.
Popper’s Falsification: verification and confirmation of theories is not
possible. We can only falsify theories. At best, theories can be
corroborated: if scientists would try to falsify a generalization, but no
scientist is able to do it, we can get confident that a generalization is
true.
Significance testing: reject or fail to reject the null hypothesis, which
describes a specific distribution of a variable of interest with a specific
mean.
Alt. hypothesis: Describes an alternative distribution of the variable of
interest if
the null hypothesis is not true. There are three forms:
specific
(H1 : a = 0.5), one-sided H1 : a > 0.5) and two sided H1 :
a 0.5).
p-value: The probability that an observed difference between a
sample
and null hypothesis came about, or a difference even
more
different from the one observed (in direction of H1 ),
under the
assumption that H0 is true.
, Cutoff value: Probability α that delineates the p-value cut-off
lines between
rejecting or not rejecting H0
Similarities Differences
1. Falsification: we can never verify a 1. Statistical data is never strictly
theory, only corroboration is possible incompatible with statistics hypothesis
at best. so falsification for certain is never
Hypothesis testing: we either reject or possible in hypothesis testing (the
fail to reject a hypothesis. dataset is a sample and NOT the
2. Both falsification and hypothesis whole population).
testing prescribe that hypothesis need 2. Statistical hypothesis are simple
to be formulated before gathering the statements, scientific theories can be
data, and the null hypothesis must be much more complex.
rejectable.
The inductive inference of hypothesis testing goes by the following steps:
if we
1. Correctly model statistical process generated by the data.
2. Perform significance testing procedure correctly.
3. Reject the null / Fail to reject the null.
4. Test is evidence against the null / We do not conclude anything.
Lecture 2: Problem of induction, Popper, and significance testing
Errors in interpreting p-values
1. “P-value is the probability that H0 is true”: False, the probability of H0
being true depends on the plausibility of H0 (e.g, the “foresight”
example), so we need more information.
2. “P is the probability that results of the trial are due to chance”: False,
non-coincidental data can generate positive p-values, depending on the
hypothesis.
For example, flipping a fair coin and getting heads in 75% of the
outcomes is 100% due to chance, but a magician who hexes the coin
such that it gives heads in 75% of the outcomes is 0% due to chance,
but the p-value is equal.
3. “P value signifies reliability: 1-P is reliability of the result”: False,
reliability depends on the prb. that H0 is true (but we do not know this)
and on size of the sample (e.g. Octopus predicting WK-results).
Statistical versus economic significance
The Cult of Statistical Significance: statistical significance is used as the
most important statistic for evaluating effects, however it tells us
nothing about the size of that effect.
2022
Lecture 1: Problem of induction, Popper, and significance testing
Econometrics: Application of statistical techniques with the purpose of
making
inferences about economic phenomena.
Skepticism about econometrics
1. Unreliable (Keynes)? In physics, we expect mathematical regularity,
economics is about social phenomena: what happens when people
interact these do not follow mathematics/statistics relationships, so
they are the wrong tool.
Is it unreliable?
Reliability: Reliability describes whether the results of an analysis
are
expected to be the same if it is repeated under the
same
circumstances.
We do not know since we use observational data to construct and test
models, since we cannot replicate the economy.
Replication crisis: Significant outcomes of experiments could not be
replicated in
different settings. Findings were largely exaggerated.
2. Trivial? We can only discover things in the data we already know from
theoretical reasoning.
Is it trivial? No, if we want to quantify certain phenomena (for
answering certain policy questions) we need to look at the data, theory
only tells us if it will happen (or not). Take e.g. the Rheinhart-Rogoff
controversy, where a mistake in a research paper led to countries
cutting government cost in 2008, believing (incorrectly) this was best
for the economy. This had disastrous consequences for a lot of people.
3. Misleading? “Lies, damned lies and statistics” fitting statistics to
your view instead of checking your view by using statistics is not
objective.
Is it misleading?
,Objectivity: Something is said to be objective when it does not
depend on
the particular characteristics of individuals who are
involved.
There are many choices: data, models, statistical methods, …. and it
turns out that econometricians come to different answers with the
same case. There are objective better and worse choices, however.
Inductive inference and the problem of induction
We have two types of data:
Observational data: no interference of the researcher in general.
Experimental data: interference of the econometrician in general.
Inductive inference: Process of drawing a general conclusion on the
basis of
observations (e.g. all pigeons can fly).
The problem of induction is that there is no certainty in general
observations (watch out for confirmation bias), but inductive reasoning
can be reliable. We should not feel certain about the knowledge we have,
but we can be more or less confident about certain things.
Popper’s Falsification: verification and confirmation of theories is not
possible. We can only falsify theories. At best, theories can be
corroborated: if scientists would try to falsify a generalization, but no
scientist is able to do it, we can get confident that a generalization is
true.
Significance testing: reject or fail to reject the null hypothesis, which
describes a specific distribution of a variable of interest with a specific
mean.
Alt. hypothesis: Describes an alternative distribution of the variable of
interest if
the null hypothesis is not true. There are three forms:
specific
(H1 : a = 0.5), one-sided H1 : a > 0.5) and two sided H1 :
a 0.5).
p-value: The probability that an observed difference between a
sample
and null hypothesis came about, or a difference even
more
different from the one observed (in direction of H1 ),
under the
assumption that H0 is true.
, Cutoff value: Probability α that delineates the p-value cut-off
lines between
rejecting or not rejecting H0
Similarities Differences
1. Falsification: we can never verify a 1. Statistical data is never strictly
theory, only corroboration is possible incompatible with statistics hypothesis
at best. so falsification for certain is never
Hypothesis testing: we either reject or possible in hypothesis testing (the
fail to reject a hypothesis. dataset is a sample and NOT the
2. Both falsification and hypothesis whole population).
testing prescribe that hypothesis need 2. Statistical hypothesis are simple
to be formulated before gathering the statements, scientific theories can be
data, and the null hypothesis must be much more complex.
rejectable.
The inductive inference of hypothesis testing goes by the following steps:
if we
1. Correctly model statistical process generated by the data.
2. Perform significance testing procedure correctly.
3. Reject the null / Fail to reject the null.
4. Test is evidence against the null / We do not conclude anything.
Lecture 2: Problem of induction, Popper, and significance testing
Errors in interpreting p-values
1. “P-value is the probability that H0 is true”: False, the probability of H0
being true depends on the plausibility of H0 (e.g, the “foresight”
example), so we need more information.
2. “P is the probability that results of the trial are due to chance”: False,
non-coincidental data can generate positive p-values, depending on the
hypothesis.
For example, flipping a fair coin and getting heads in 75% of the
outcomes is 100% due to chance, but a magician who hexes the coin
such that it gives heads in 75% of the outcomes is 0% due to chance,
but the p-value is equal.
3. “P value signifies reliability: 1-P is reliability of the result”: False,
reliability depends on the prb. that H0 is true (but we do not know this)
and on size of the sample (e.g. Octopus predicting WK-results).
Statistical versus economic significance
The Cult of Statistical Significance: statistical significance is used as the
most important statistic for evaluating effects, however it tells us
nothing about the size of that effect.
