Research Methods 2.0 B
INDEX
1. Responsibilities for Scientists 2
2. Literature Reviewing 5
3. External Validity 9
4. Field Research 12
5. Single-case Research 16
6. Qualitative Research 18
This summary includes (almost) everything from the lectures 8 until 11, from the book chapters and the
lectures.
DISCLAIMER
This summary is made by a student!
Studying from it and relying on it for 100% is your own responsibility.
THANKS & GOOD LUCK!!! J
J YOU CAN DO IT !!!!
, 2
Responsibilities for Scientists (Kite & Whitley, Ch. 4)
Mistakes and Errors in Research
- Of course everyone sometimes makes errors: have to distinguish mistakes from culpable error
o Culpable error is ethically unacceptable!!! (example = systematic carelessness)
- Culpable error = unethical because of the consequences:
o Harm to research participants. 2 types
§ Wasting the participant’s time because they are there for no reason
§ Can lead to direct harm (e.g. physical) if the process is not designed correctly and
with care
o Harm to science: if results of poor research find their way into science à false info & false
conclusions (e.g. results of autism & vaccinations study)
o Harm to public: if these results find way to public, can be even more harmful, as also seen
with the results of the autism & vaccinations study!!
What are culpable errors?
- Incompetence and negligence
o Usually incompetence affects research as “lack of ability to design internally valid research”
o Can take 2 additional forms
§ Subject matter incompetence – lack of expertise in the research area investigators
are examining à will be unable to form research questions and hypotheses of
scientific value
§ Statistical matter incompetence – lack of expertise in the area of statistical testing
à also affects whether you find e.g. significant results
- Questionable research practices
o Questionable research practices are decisions researchers make when planning a study or
during and after data collection that can lead to erroneous or misleading results
§ Not considered unethical because there is no intention to deceive, but researchers
might “cut corners” in order to get significant results because career depends on it
- Questionable research practices during data collection, interpretation and reporting
Term Definition
During data collection and analysis (= data abuse)
Data dredging Collecting data on a large number of variables but focusing only on the
statistically significant results among them (shotgun research)
Data snooping Periodically checking results for statistical significance during data collection and
stopping data collection when statistically significant results are found
Data trimming Selectively discarding data until results become statistically significant
Data torturing Selectively including or excluding potential moderator variables until statistical
significance is found
Methodological tuning Tweaking the study methodology until statistically significant results are found
During data interpretation and reporting
Accentuating the positive Has two aspects
- Being more critical of results that are inconsistent with one’s hypotheses
than of results that are consistent with the hypotheses
- Focusing one’s attention on statistically significant findings while
ignoring nonsignificant findings
HARKing Formulating hypotheses after analyses but treating them as a priori hypotheses
when reporting results
- Correcting mistakes and errors
o When finding mistakes in your own/someone else’s work, are obliged to correct them
o Several ways to do it
§ If new research/data refutes theory/results from previous work, the error is
automatically “corrected”
, 3
§ If mistake is found by the writer after publishing à ask journal to print correction or,
if error makes entire study invalid, retract the paper
§ If mistake is found by reader after publishing à write critique to journal à is
reviewed, maybe kept the same or changed/retracted
Scientific misconduct – additional culpable errors
- Even though most errors are due to mistakes, sometimes people do make intentional mistakes to
tweak outcomes etc.
- Data falsification: Data forgery – report the results of experiments that were never conducted
o “easy” à short term gains for researchers. They do not have to take the time and money to
collect all data etc., they just make up results. But once discovered, short term gain
becomes a career killer.
o But can go much further than consequences for the individual researcher!!! E.g. Breuning
published reports on effectiveness of drug treatments for children who showed self-
injurious forms of behavior, à said side effects were very severe à children were taken off
of medications à these “findings” even changed some drug-policies in the US!! Until they
discovered the studies he talked about were actually never conducted
- Preventing scientific misconduct
o Studies have to go past institutional review boards
o Other ways is e.g. requiring researchers to complete training on the responsible conduct of
research. + emphasize QUALITY over QUANTITY of publications in university tenure and in
promotion processes and in the criteria established for research excellence awards
Ethical issues in publication – additional culpable errors
- Authorship credit
o Order of authors implies degree of contribution, question of whether person receives
authorship credit & the order speaks to the issue of the fair allocation of credit for the work
§ APA has guidelines for this, but still things like “substantial scientific contributions”
à vague!
§ Winston & Kosslyn: more structure approaches, e.g. developing research ide &
writing manuscript are each worth 50 points, whereas measure selection is worth 10
points, etc. à no. of points decides order
- Duplicate publication – publishing the same work in different journals
o Distorts the scientific knowledge base by making it seem like there is more info available on
1 topic than is the actual truth
o 3 exceptions of prohibition against duplicate publication: always refer to original publication
§ It’s ok to present a study at a convention & publish it in a journal
§ Taking a technical article and rewriting it for a nontechnical outlet
§ Write a report of previously published material for a journal/edited book bc editor
thinks info is useful to readers who may not have access to the original publication
o Piecemeal publication – taking pieces of original dataset for separate publications
§ Is allowed e.g. if data for 2 different studies are collected together
- Plagiarism – the act of taking someone else’s work or ideas and passing them off as your own
o Avoiding plagiarism
§ Carefully keep track of sources you take information from
§ Made difficult by two factors
• Fair use doctrine: you’re allowed to use copyrighted material in a reasonable
manner w/o permission of the copyright holder
• You are allowed to use common knowledge/widely known facts – do not
need citations for them
INDEX
1. Responsibilities for Scientists 2
2. Literature Reviewing 5
3. External Validity 9
4. Field Research 12
5. Single-case Research 16
6. Qualitative Research 18
This summary includes (almost) everything from the lectures 8 until 11, from the book chapters and the
lectures.
DISCLAIMER
This summary is made by a student!
Studying from it and relying on it for 100% is your own responsibility.
THANKS & GOOD LUCK!!! J
J YOU CAN DO IT !!!!
, 2
Responsibilities for Scientists (Kite & Whitley, Ch. 4)
Mistakes and Errors in Research
- Of course everyone sometimes makes errors: have to distinguish mistakes from culpable error
o Culpable error is ethically unacceptable!!! (example = systematic carelessness)
- Culpable error = unethical because of the consequences:
o Harm to research participants. 2 types
§ Wasting the participant’s time because they are there for no reason
§ Can lead to direct harm (e.g. physical) if the process is not designed correctly and
with care
o Harm to science: if results of poor research find their way into science à false info & false
conclusions (e.g. results of autism & vaccinations study)
o Harm to public: if these results find way to public, can be even more harmful, as also seen
with the results of the autism & vaccinations study!!
What are culpable errors?
- Incompetence and negligence
o Usually incompetence affects research as “lack of ability to design internally valid research”
o Can take 2 additional forms
§ Subject matter incompetence – lack of expertise in the research area investigators
are examining à will be unable to form research questions and hypotheses of
scientific value
§ Statistical matter incompetence – lack of expertise in the area of statistical testing
à also affects whether you find e.g. significant results
- Questionable research practices
o Questionable research practices are decisions researchers make when planning a study or
during and after data collection that can lead to erroneous or misleading results
§ Not considered unethical because there is no intention to deceive, but researchers
might “cut corners” in order to get significant results because career depends on it
- Questionable research practices during data collection, interpretation and reporting
Term Definition
During data collection and analysis (= data abuse)
Data dredging Collecting data on a large number of variables but focusing only on the
statistically significant results among them (shotgun research)
Data snooping Periodically checking results for statistical significance during data collection and
stopping data collection when statistically significant results are found
Data trimming Selectively discarding data until results become statistically significant
Data torturing Selectively including or excluding potential moderator variables until statistical
significance is found
Methodological tuning Tweaking the study methodology until statistically significant results are found
During data interpretation and reporting
Accentuating the positive Has two aspects
- Being more critical of results that are inconsistent with one’s hypotheses
than of results that are consistent with the hypotheses
- Focusing one’s attention on statistically significant findings while
ignoring nonsignificant findings
HARKing Formulating hypotheses after analyses but treating them as a priori hypotheses
when reporting results
- Correcting mistakes and errors
o When finding mistakes in your own/someone else’s work, are obliged to correct them
o Several ways to do it
§ If new research/data refutes theory/results from previous work, the error is
automatically “corrected”
, 3
§ If mistake is found by the writer after publishing à ask journal to print correction or,
if error makes entire study invalid, retract the paper
§ If mistake is found by reader after publishing à write critique to journal à is
reviewed, maybe kept the same or changed/retracted
Scientific misconduct – additional culpable errors
- Even though most errors are due to mistakes, sometimes people do make intentional mistakes to
tweak outcomes etc.
- Data falsification: Data forgery – report the results of experiments that were never conducted
o “easy” à short term gains for researchers. They do not have to take the time and money to
collect all data etc., they just make up results. But once discovered, short term gain
becomes a career killer.
o But can go much further than consequences for the individual researcher!!! E.g. Breuning
published reports on effectiveness of drug treatments for children who showed self-
injurious forms of behavior, à said side effects were very severe à children were taken off
of medications à these “findings” even changed some drug-policies in the US!! Until they
discovered the studies he talked about were actually never conducted
- Preventing scientific misconduct
o Studies have to go past institutional review boards
o Other ways is e.g. requiring researchers to complete training on the responsible conduct of
research. + emphasize QUALITY over QUANTITY of publications in university tenure and in
promotion processes and in the criteria established for research excellence awards
Ethical issues in publication – additional culpable errors
- Authorship credit
o Order of authors implies degree of contribution, question of whether person receives
authorship credit & the order speaks to the issue of the fair allocation of credit for the work
§ APA has guidelines for this, but still things like “substantial scientific contributions”
à vague!
§ Winston & Kosslyn: more structure approaches, e.g. developing research ide &
writing manuscript are each worth 50 points, whereas measure selection is worth 10
points, etc. à no. of points decides order
- Duplicate publication – publishing the same work in different journals
o Distorts the scientific knowledge base by making it seem like there is more info available on
1 topic than is the actual truth
o 3 exceptions of prohibition against duplicate publication: always refer to original publication
§ It’s ok to present a study at a convention & publish it in a journal
§ Taking a technical article and rewriting it for a nontechnical outlet
§ Write a report of previously published material for a journal/edited book bc editor
thinks info is useful to readers who may not have access to the original publication
o Piecemeal publication – taking pieces of original dataset for separate publications
§ Is allowed e.g. if data for 2 different studies are collected together
- Plagiarism – the act of taking someone else’s work or ideas and passing them off as your own
o Avoiding plagiarism
§ Carefully keep track of sources you take information from
§ Made difficult by two factors
• Fair use doctrine: you’re allowed to use copyrighted material in a reasonable
manner w/o permission of the copyright holder
• You are allowed to use common knowledge/widely known facts – do not
need citations for them
