Summary Statistics 1B

Statistic 1b 6.3

Use and Abuse of Tests:

 Each test is valid only in certain circumstances
 AH: there is some effect or difference
 NH: there is no difference
 A low p-value presents good evidence that the research
hypothesis is true
 The spirit of a test of significance is to give a clear statement
of the degree of evidence provided by the sample against the
null hypothesis
 Statistically significant P ≤ 0.05
 However, there is no sharp boarder b/w “significant” and “not
significant” only increasingly strong evidence as the P-value
decreases
 “a scientific fact should be regarded as experimentally
established only if a properly designed experiment rarely fails
to give this level of significance.”
 When large samples are available, even tiny deviations from
the null hypothesis will be significant
 ∝ =0.05 good evidence that an effect is present
 statistical significance is not the same as practical significance
 statistical significance rarely tells us about the importance of
the experimental results
 “the foolish user of statistics who feed the data to a computer
without explanatory analysis will often be embarrassed”
o plot your data and examine it carefully, beware if
outliers
 “absent of evidence is not evidence of absence!”
 another important aspect of planning a study is to verify that
the test you plan to use does have high probability of
detecting an effect of the size you hope to find
 tests of significance and confidence intervals are based on the
laws of probability
o randomization in sampling or experimentation ensures
that these laws apply
 BUT we must often analyze data that do not arise from
randomized samples or experiments.
o To apply statistical inference to such data, we must have
confidence in a probability model for the data
 The reasoning behind statistical significance works well if you
decide what effect you are seeking, design an experiment or
sample to search for it, and use a test of significance to weigh
the evidence you get
 Many important discoveries have been made by accident
rather than by design

,  You cannot legitimately test a hypothesis on the same data
that first suggested that hypothesis
 P-values are more informative than the reject-or-not result
 Very small p-values can be highly significant, especially when
based on a large sample
 Lack of significance does not imply that the NH is true,
especially when the test has low power
Significance tests are not always valid! – faulty data collection,
outliers

Statistic Lecture 17

 CI – values which we cannot reject
 Almost always we have to use two-tailed tests
 Errors
o You cannot argue for the NH when you just cannot reject
it
o You can only argue about difference if you tested for the
difference
 Fix the sample size before the experiment!
 Statistical significance does not imply practical significance!
 Decide which type of test you use before looking at the data!
 Stroop effect:
 Ways to summarize and report results:
o Make a plot
o Absolute effect e.g.
o CI on the effect e.g.
o Statistical significance e.g.
o Standardized effect size – is an effect size scaled by the
amount of error in the data; this makes all effects
comparable, even ones with different units
 Against a standard unit e.g. standard deviation
 Cohen’s d
 d=
 signal-to-noise ratio
 estimate of
 Q – “How many population standard
deviation is my effect?”
 Quantifies the effect size, does not depend
on N; z-score does depend on N
 d has no units
 d doesn’t change when units are changed
 d is unaffected by linear transformation
 Rule of thumb:
 |d| ≈ 0.2 → small effect
 |d| ≈ 0.5 → medium effect
 |d| ≈ 0.8 → large effect