PROBLEM 7. PSYCHOLOGICAL RESEARCH 1
Significance of different kinds of research by Dr. A.D. De Groot
Empirical cycle > observation – induction – deduction – testing
(observe – guess – predict – check)
Hypothesis Testing Research: one assesses whether the
consequences/predictions derived from one or more hypothesis occur.
→ Precise formulation of hypothesis, as objective testing
procedures as possible/ and specified in advance.
→ Applies to deduction and testing of scientific cycle
Hypothesis testing Material-exploration Research: does not test hypothesis, generates
VS hypotheses/theories or interprets the available material.
Material-exploration → Try to trace associations (Interesting material, let’s find sth)
→ General intention specified beforehand but the processing
(scoring methods and associations) steps are based on (at
least partly) ad-hoc decisions.
Mixed research: involves hypothesis testing, involves also material-
exploration
→ Still possible and desirable to discriminate the parts.
→ Different scientific value, thought process and certainty.
SRS from population
Predefined hypothesis, clear, sufficient and precise
Hypothesis testing
Positive result- probability lower than α predetermined threshold.
Single
→ Problems: multiple hypothesis can be present and researched
simultaneously, elements of material-exploration type.
The probability of all N (null hypothesis) being falsely rejected equals
alpha but that’s not the case for one or two hypothesis not selected in
the beginning.
n out of N significant > cannot rely on these positive results
> control that by testing n compared to αNN
Hypothesis testing
From 3 onwards > rejecting null hypothesis, all three not by
Multiple
chance but not sure for two or one of them alone.
Serious weakening of the argument from significance in case n is
small relative to N
Reason: evaluation of the outcomes of the statistical tests is
preceded by a selection, based on those same outcomes.
Material-exploration Exploration of available material under different perspectives and
ways that has not been prespecified with the aim of finding
associations or seek confirmation.
Extract also things, which are in there accidentally.
→ Trying: experiments with several variables, several operations
for the same variable, subgroupings, associations, statistical
tests etc.
→ Selecting: executes only methods that promise something, ad
hoc selection
N becomes undetermined > exact interpretation of the meaning
, PROBLEM 7. PSYCHOLOGICAL RESEARCH 2
of “positive results” is no longer possible.
Prohibition not to calculations but to interpretation of the outcomes.
Exploratory research does not have an impact, mainly focused on
hypothesis generation
The errors of irresponsible conclusions are possible in qualitative
Conclusions
and quantitative research – ad hoc associations
→ Quantitative research, errors in reasoning and
irresponsible conclusions are eventually demonstrable
“lies – damned lies –ad hoc interpretations”
An Agenda for Purely Confirmatory Research
Confirmation bias: operates in three ways
1. Ambiguous info interpreted accordingly to prior beliefs
2. Search for info that confirms rather than falsifies their hypothesis
3. Easier remembering of supporting evidence to their position
Hindsight bias: tendency to judge an event as more predictable after it has occurred
Reasons: the pressure increases when career is dependent on how many publications you have
Outcomes: published results may be false, because of dubious/inappropriate methods of
observation, analysis, and reporting
Fairy-tale factor: researchers do not commit themselves to a plan of analysis before they see
the data > fine tuning of data
Increased probability of fictional findings and reduced replicability
Almost no psychological research is conducted in a purely
confirmatory fashion > freedom of analysis, tricks to obtain the desired
results
Cherry picking: measure many variables and choose the ones
that provide desirable outcomes
Exploring various data transformations: one-sided p’s and post
hoc hypotheses to fit the data (easier with software)
→ Torture data until they confess, not necessary bad but it
should be clearly acknowledged
Bad science → Use of data one time, not explore and then hypothesize on
the same data
Figure of science: from exploratory research of data, formation of
hypothesis >>> purely confirmatory, analysis specified before testing.
Where a study falls in depends on
Initial outcome (bad > exploratory)
Clarity of research question (vague > exploratory)
Amount of data (more variables > exploratory)
Priori beliefs (strong belief > more exploration)
NO disapproval of exploratory research, as long as it is acknowledged
Good science Separation of explanatory and confirmatory experiments
1st Stage: Exploratory research of data as they please, not strong
evidence but basis for future research
Significance of different kinds of research by Dr. A.D. De Groot
Empirical cycle > observation – induction – deduction – testing
(observe – guess – predict – check)
Hypothesis Testing Research: one assesses whether the
consequences/predictions derived from one or more hypothesis occur.
→ Precise formulation of hypothesis, as objective testing
procedures as possible/ and specified in advance.
→ Applies to deduction and testing of scientific cycle
Hypothesis testing Material-exploration Research: does not test hypothesis, generates
VS hypotheses/theories or interprets the available material.
Material-exploration → Try to trace associations (Interesting material, let’s find sth)
→ General intention specified beforehand but the processing
(scoring methods and associations) steps are based on (at
least partly) ad-hoc decisions.
Mixed research: involves hypothesis testing, involves also material-
exploration
→ Still possible and desirable to discriminate the parts.
→ Different scientific value, thought process and certainty.
SRS from population
Predefined hypothesis, clear, sufficient and precise
Hypothesis testing
Positive result- probability lower than α predetermined threshold.
Single
→ Problems: multiple hypothesis can be present and researched
simultaneously, elements of material-exploration type.
The probability of all N (null hypothesis) being falsely rejected equals
alpha but that’s not the case for one or two hypothesis not selected in
the beginning.
n out of N significant > cannot rely on these positive results
> control that by testing n compared to αNN
Hypothesis testing
From 3 onwards > rejecting null hypothesis, all three not by
Multiple
chance but not sure for two or one of them alone.
Serious weakening of the argument from significance in case n is
small relative to N
Reason: evaluation of the outcomes of the statistical tests is
preceded by a selection, based on those same outcomes.
Material-exploration Exploration of available material under different perspectives and
ways that has not been prespecified with the aim of finding
associations or seek confirmation.
Extract also things, which are in there accidentally.
→ Trying: experiments with several variables, several operations
for the same variable, subgroupings, associations, statistical
tests etc.
→ Selecting: executes only methods that promise something, ad
hoc selection
N becomes undetermined > exact interpretation of the meaning
, PROBLEM 7. PSYCHOLOGICAL RESEARCH 2
of “positive results” is no longer possible.
Prohibition not to calculations but to interpretation of the outcomes.
Exploratory research does not have an impact, mainly focused on
hypothesis generation
The errors of irresponsible conclusions are possible in qualitative
Conclusions
and quantitative research – ad hoc associations
→ Quantitative research, errors in reasoning and
irresponsible conclusions are eventually demonstrable
“lies – damned lies –ad hoc interpretations”
An Agenda for Purely Confirmatory Research
Confirmation bias: operates in three ways
1. Ambiguous info interpreted accordingly to prior beliefs
2. Search for info that confirms rather than falsifies their hypothesis
3. Easier remembering of supporting evidence to their position
Hindsight bias: tendency to judge an event as more predictable after it has occurred
Reasons: the pressure increases when career is dependent on how many publications you have
Outcomes: published results may be false, because of dubious/inappropriate methods of
observation, analysis, and reporting
Fairy-tale factor: researchers do not commit themselves to a plan of analysis before they see
the data > fine tuning of data
Increased probability of fictional findings and reduced replicability
Almost no psychological research is conducted in a purely
confirmatory fashion > freedom of analysis, tricks to obtain the desired
results
Cherry picking: measure many variables and choose the ones
that provide desirable outcomes
Exploring various data transformations: one-sided p’s and post
hoc hypotheses to fit the data (easier with software)
→ Torture data until they confess, not necessary bad but it
should be clearly acknowledged
Bad science → Use of data one time, not explore and then hypothesize on
the same data
Figure of science: from exploratory research of data, formation of
hypothesis >>> purely confirmatory, analysis specified before testing.
Where a study falls in depends on
Initial outcome (bad > exploratory)
Clarity of research question (vague > exploratory)
Amount of data (more variables > exploratory)
Priori beliefs (strong belief > more exploration)
NO disapproval of exploratory research, as long as it is acknowledged
Good science Separation of explanatory and confirmatory experiments
1st Stage: Exploratory research of data as they please, not strong
evidence but basis for future research
