Research Methods in Clinical Neuropsychology
Introductory test:
1. Name two advantages of effect sizes over significance tests?
• Significance depends on sample size, ES not
• ES helps to compare significant effects with each other
2. Your study is underpowered! What does it mean in plain language?
• Given patients with dementia are really impaired in prospective memory, it is unlikely that
you will reveal this effect in your study
3. Name two different active control group designs!
• Dose control (the same but less intensive treatment)
• Dismantling design (same treatment, but one element of your treatment is taken out)
• These designs are to find out what is really responsible for your treatment effects
4. How to report the validity of a diagnostic test?
• Overall diagnostic accuracy (e.g. AUC = area under the curve)
• Sensitivity towards disease (classify the ones being sick), specificity towards no disease
(classify the ones not having it).
5. Name two single case research designs!
• ABAB design
• Multiple baseline design
6. What is the Reliable Change Index?
• Indicates the change in a measure from one time to another (e.g. before and after
treatment), taking into consideration the variability of scores at the first time, as well as the
test-retest reliability
7. How does a Delphi methodology contribute to reaching consensus?
• A Delphi methodology includes a group of experts on a topic (the expert panel) and
facilitates collective decision making by a structured and anonymous way of communication
Lecture 1A. Basic study designs:
Designing, implementing, analyzing
Group designs: Selection, recruitment,
measurement
Group designs: >>
- Retrospective (asking how the
patient was doing before the
research)
- Case-control or cross sectional:
for each patient is one control
person (similar in characteristics,
background information).
Selection, recruitment, measurement:
- Internal validity: our planned
research has not been performed
properly, mistakes, dropouts.
,Selection and recruitment
➢ Establishing selection criteria
Inclusion criteria:
- Demographic characteristics (65+ or below 65)
- Clinical characteristics
- Geographic characteristics
- Temporal characteristics
Exclusion criteria:
- Risk of being lost at follow up
- Inability to provide good data
- At risk for possible adverse effects
- Non-representative for population
➢ Sampling
Nonprobability samples:
- Convenience samples (mostly done, just take who you can get)
- Snowball sampling
Probability sample:
- Simple random samples (randomly picking)
- Systematic random sample (randomly picking but according to a certain system; every other
person)
- Stratified random sample (certain equal distributions; 50 males 50 females, or ages equally
represented)
- Cluster sample (for example 3 out of 20 clinics are randomly picked, but all patients from
those clinics are used)
Random sample: sounds perfect, but is almost not possible in clinical research (convenience
sample is a selective group of people, always consider this in your discussion)
Measurement:
➢ Scaling
Categorical or continuous variables?
- Continuous variables contain more information, are more flexible, and often preferred.
- Categorical is straight forward: it’s clear, someone can work or not. Yes or no.
➢ Precision (reliability)
- Degree to which individuals retain relative position within distribution of scores from one
testing session to another (same ranking, relative to other people)
- Most often presented as correlation in test-retest reliability (stability)
- People with disorders are typically more fluctuating than the general population (healthy
people)
,➢ Accuracy (validity)
- Good: groups that assume to be having attention problems should be worse than a healthy
group. If they are not, then the test is not really measuring the right thing >> not accurate.
- Discriminative validity: instrument’s ability to distinguish between relevant participant
groups.
- Incremental validity: how it works together with other measures of similar constructs.
➢ Sensitivity: Sensitive measure required that detects improvements even in healthy people!
➢ How to reduce random error
- Standardizing the measurement
- Training the staff doing assessments
- Automating the instrument
- Blinding
- Repeating the measurement
➢ Modern assessment methods: Computerized or paper-pencil assessment?
Reasons that may explain the reluctance:
1. Psychometric obstacles
- Reliability of traditional and computerized tests
- Equivalence of computer tests and paper-and-pencil tests
- Quality of normative data
- Norms are not valid anymore: it’s easier to just stick to the paper-pencil test then.
2. Technical obstacles
- E.g. speed in technical developments hamper work on psychometric properties
3. Theoretical obstacles
- Extensiveness of the body of knowledge
- Theoretical paradigms and their practical value
4. Strategic obstacles
- Rapidly growing number of new approaches results in incomparability of results
➢ Some ideas about the future of neuropsychological testing
- Advantages:
o save time
o More accurate according to scoring
o Cheaper, it can be done by an assistant instead of psychologist self.
o Classic scoring of trail-making test does not distinguish between what the person is
actually doing, sloppy worker with many mistakes or very accurate but slow worker.
- Disadvantage:
o lose information by not observing how people do the test etc.
- Computerized adaptive testing
o Gain in efficiency (time and precision) by selecting next item based on performance on
current item.
- Nominal response model
o Different meanings of different types of errors (e.g. response style on TMT)
- Test linking
o Direct comparison of tests and integration into individual report (e.g. ANDI.nl)
- Person fit statistics
o Performance validity based on fit of one item relative to overall performance
- Web-based testing, mobile platforms, wearables
, o Assessment goes out of clinic into “real life”, increase in longitudinal repeated
assessments
Lecture 1B. Pitfalls in significance testing
➢ Post-hoc testing
Principle of a randomized
controlled trial:
No hypotheses of group differences on demographic variables? >> No testing necessary
➢ Cherry picking = focusing on the results that fit you
➢ Nonspecific hypotheses:
- Hypothesis: ‘Mental disorders have a
genetic underpinning‘
>> Correlation of 1000 gene loci
with 10 mental disorders
>> 10.000 possible specific hypothesis
- Problem in α-error adjustment!
o α-error correction by .05/10.000
o Controls for α-error inflation, but drastically reduces power!
- Better: Stepwise hypotheses generating and testing
o Exploratory (hypothesis-generating) study: No significance testing
o Hypothesis testing on independent data set (on specific hypothesis)
➢ Multiple testing
- One hypothesis should NOT be examined with multiple tests
- Introduction of multiple hypotheses is possible
o Should be conceptually independent
o Each hypothesis treated as if tested in a different study
Introductory test:
1. Name two advantages of effect sizes over significance tests?
• Significance depends on sample size, ES not
• ES helps to compare significant effects with each other
2. Your study is underpowered! What does it mean in plain language?
• Given patients with dementia are really impaired in prospective memory, it is unlikely that
you will reveal this effect in your study
3. Name two different active control group designs!
• Dose control (the same but less intensive treatment)
• Dismantling design (same treatment, but one element of your treatment is taken out)
• These designs are to find out what is really responsible for your treatment effects
4. How to report the validity of a diagnostic test?
• Overall diagnostic accuracy (e.g. AUC = area under the curve)
• Sensitivity towards disease (classify the ones being sick), specificity towards no disease
(classify the ones not having it).
5. Name two single case research designs!
• ABAB design
• Multiple baseline design
6. What is the Reliable Change Index?
• Indicates the change in a measure from one time to another (e.g. before and after
treatment), taking into consideration the variability of scores at the first time, as well as the
test-retest reliability
7. How does a Delphi methodology contribute to reaching consensus?
• A Delphi methodology includes a group of experts on a topic (the expert panel) and
facilitates collective decision making by a structured and anonymous way of communication
Lecture 1A. Basic study designs:
Designing, implementing, analyzing
Group designs: Selection, recruitment,
measurement
Group designs: >>
- Retrospective (asking how the
patient was doing before the
research)
- Case-control or cross sectional:
for each patient is one control
person (similar in characteristics,
background information).
Selection, recruitment, measurement:
- Internal validity: our planned
research has not been performed
properly, mistakes, dropouts.
,Selection and recruitment
➢ Establishing selection criteria
Inclusion criteria:
- Demographic characteristics (65+ or below 65)
- Clinical characteristics
- Geographic characteristics
- Temporal characteristics
Exclusion criteria:
- Risk of being lost at follow up
- Inability to provide good data
- At risk for possible adverse effects
- Non-representative for population
➢ Sampling
Nonprobability samples:
- Convenience samples (mostly done, just take who you can get)
- Snowball sampling
Probability sample:
- Simple random samples (randomly picking)
- Systematic random sample (randomly picking but according to a certain system; every other
person)
- Stratified random sample (certain equal distributions; 50 males 50 females, or ages equally
represented)
- Cluster sample (for example 3 out of 20 clinics are randomly picked, but all patients from
those clinics are used)
Random sample: sounds perfect, but is almost not possible in clinical research (convenience
sample is a selective group of people, always consider this in your discussion)
Measurement:
➢ Scaling
Categorical or continuous variables?
- Continuous variables contain more information, are more flexible, and often preferred.
- Categorical is straight forward: it’s clear, someone can work or not. Yes or no.
➢ Precision (reliability)
- Degree to which individuals retain relative position within distribution of scores from one
testing session to another (same ranking, relative to other people)
- Most often presented as correlation in test-retest reliability (stability)
- People with disorders are typically more fluctuating than the general population (healthy
people)
,➢ Accuracy (validity)
- Good: groups that assume to be having attention problems should be worse than a healthy
group. If they are not, then the test is not really measuring the right thing >> not accurate.
- Discriminative validity: instrument’s ability to distinguish between relevant participant
groups.
- Incremental validity: how it works together with other measures of similar constructs.
➢ Sensitivity: Sensitive measure required that detects improvements even in healthy people!
➢ How to reduce random error
- Standardizing the measurement
- Training the staff doing assessments
- Automating the instrument
- Blinding
- Repeating the measurement
➢ Modern assessment methods: Computerized or paper-pencil assessment?
Reasons that may explain the reluctance:
1. Psychometric obstacles
- Reliability of traditional and computerized tests
- Equivalence of computer tests and paper-and-pencil tests
- Quality of normative data
- Norms are not valid anymore: it’s easier to just stick to the paper-pencil test then.
2. Technical obstacles
- E.g. speed in technical developments hamper work on psychometric properties
3. Theoretical obstacles
- Extensiveness of the body of knowledge
- Theoretical paradigms and their practical value
4. Strategic obstacles
- Rapidly growing number of new approaches results in incomparability of results
➢ Some ideas about the future of neuropsychological testing
- Advantages:
o save time
o More accurate according to scoring
o Cheaper, it can be done by an assistant instead of psychologist self.
o Classic scoring of trail-making test does not distinguish between what the person is
actually doing, sloppy worker with many mistakes or very accurate but slow worker.
- Disadvantage:
o lose information by not observing how people do the test etc.
- Computerized adaptive testing
o Gain in efficiency (time and precision) by selecting next item based on performance on
current item.
- Nominal response model
o Different meanings of different types of errors (e.g. response style on TMT)
- Test linking
o Direct comparison of tests and integration into individual report (e.g. ANDI.nl)
- Person fit statistics
o Performance validity based on fit of one item relative to overall performance
- Web-based testing, mobile platforms, wearables
, o Assessment goes out of clinic into “real life”, increase in longitudinal repeated
assessments
Lecture 1B. Pitfalls in significance testing
➢ Post-hoc testing
Principle of a randomized
controlled trial:
No hypotheses of group differences on demographic variables? >> No testing necessary
➢ Cherry picking = focusing on the results that fit you
➢ Nonspecific hypotheses:
- Hypothesis: ‘Mental disorders have a
genetic underpinning‘
>> Correlation of 1000 gene loci
with 10 mental disorders
>> 10.000 possible specific hypothesis
- Problem in α-error adjustment!
o α-error correction by .05/10.000
o Controls for α-error inflation, but drastically reduces power!
- Better: Stepwise hypotheses generating and testing
o Exploratory (hypothesis-generating) study: No significance testing
o Hypothesis testing on independent data set (on specific hypothesis)
➢ Multiple testing
- One hypothesis should NOT be examined with multiple tests
- Introduction of multiple hypotheses is possible
o Should be conceptually independent
o Each hypothesis treated as if tested in a different study
