Research Methods in Clinical Neuropsychology
Learning goals – understand:
- Basic study designs, such as cross-sectional studies, case control studies, and cohort studies.
- Randomized Controlled Trials (e.g., following CONSORT statement)
o Classic state of intervention to find out if the intervention is working or not
- Single case research designs: Design, application, and analysis.
- Ethical considerations and research ethics in clinical neuropsychology.
- Pitfalls in the analysis of quasi-experimental clinical studies.
o With patients it is difficult to conclude causal effect ‘this is caused by’
o Taking the group as they are: brain tumor is not manipulated
o Sometimes overstep the data with correlational data
- Evaluating treatment efficacy: Consequences of expected effect size.
o P-value is the starting point, but actually you could skip it. But due the expectations
from the research field, you report it Effect Size (ES) is more important
- Clinical decision making using diagnostic tests.
- Reevaluation of patients with neuropsychological impairments.
- Selection of intervention methods: Understanding meta-analyses.
Do not:
- Study calculations or statistical output.
- Study formulas → these are only used for understanding the concepts. You should be able to
explain how these constructs are determined, which can be done by referring to formulas.
You can also explain in words how certain terms are determined.
Course literature
- Book chapters.
- Research articles.
Exam
- 6 open questions on various topics of the course (each topic devoted to one questions which
also contains sub-questions)
- Maximum of 60 points.
- Answers in EN or NL
,Week 1 – Lecture 15 November – Introduction lecture
Introductory test
Effect Size (ES) = the size of the impairment
• Difficult to get significant effect in small groups, but ES can still conclude how large an effect
is, independently of the sample size (because ES is not rely on sample size)
• In large samples: small effects get significant
• ES helps to compare significant effects with each other
o Don’t compare if you only at the significance
Underpowered = Lower likelihood of finding a true existing effect. If there is an effect, you have too
few participants to find it. Assuming that there is an effect. Sample not big enough to show it and
miss true effect.
Active control group design = Control group also receives the treatment. If our treatment works for a
group, compared to a control group
• Dose control: same substance but lower dose in pharmacology therapy
• Dismantling design: complex treatment (a lot of steps in assessment/treatment). We do
exactly the same, we take out one active step (don’t give feedback on the results OR remove
cognitive training)
• Waitlist control group: do nothing but will promised to receive the treatment later
o Treatment group compared to nothing is not what want to achieve with treatment
results
Mild cognitive impairment (MCI). Which statistical information do you describe for the test validity?
• Overall diagnostic accuracy (AUC)
• Spectivity: towards no disease
• Sensitivity: towards disease
→ how often do I commit misclassification
Explore effectiveness of treatment on single patient/ single case studies
• ABAB design: give treatment and return to baseline
• Multiple baseline design: with single patients
Reliable Change Index: indicates the change in a measure from one time to another (before and
after treatment) ON A SINGLE PERSON. Taking into consideration the variability of scores at the first
time, learning effects, practice effects, as well as the test-retest reliability
• Current state is different than in first assessment
o Negative score: with progressive dementia it further declines 2nd assessment
compared to 1st assessment
Delphi Methodology = includes a group of experts on a topic (the expert panel) and facilitates
collective decision making by a structured and anonymous way of communication
→ Exam: not ask for definitions. But askes to give examples, (dis)advantages
,Group design
Same time point
• Cross Sectional: one time point right now (patient and control sample) and compare them
(patient and control sample)
• Case-control: for each patient, you select a control person (same age, gender, educational
level). So for each patient a single matching control person.
o But most time this is not happening on individual level, but on group level
More time points
• Longitudinal study: follow patients over time
• Prospective: now recruitment and see how develop control
• Retrospective: talk to school, friends, parents and look at time BEFORE assessment.
o Bit more difficult
Aim of research is real truth (in world) how patients with X (dementia) will perform in Y (prospective
memory).
Real truth is not possible, because we cannot take all dementia patients as participants. Therefore do
an assessment to come close to the truth, by selection that will represent the population.
External validity = Phenomena of interest on intended variables. ULTIMATE GOAL
• How people struggle and test in REAL life. Really important because we don’t do
experimental research. But we want to know how patients perform, learn and struggle in life.
Test result is approximately, and not the real situation.
• Variable is hopefully represent the phenomena of interests/target population
Internal validity = Intended sample on actual subjects.
, • When I assess my people, I make certain mistakes (confounding variables, computer is not
working, not understanding instructions, mistaking in time measurements) and may deviate
from plan.
• Experimental psychologists to take care for low mistakes as possible by protocols, lab
Selection and recruitment = to reach a represented true population in sample
• Inclusion: demographic characteristics, clinical characteristics, geopathic, temporal
• Exclusion: risk of begin lost at follow up, inability to provide good date, at risk possible
adverse effects, non-representative for population
→ Aim: to represent the population with the right characteristics
Sampling: how do I select people
• Non-probability samples
o Convenience samples: people who are accessible to me, interested to take
part, approaching the network. Most studies are convenience samples.
▪ Difficult to interpret findings and raw conclusions for general population.
Because it depends on collaboration of participants → interpretations
drawing with care
• Snowball sampling
• Probability sample (idea situation: random select, people willing to participate, have time,
motivation)
• Simple random sampling: randomly drawn from population
• Systematic random sample: not completely random, but in each age cohort random (20-
30, 30-40, 40-50 etc)
• Stratified random sample: divides population into smaller subgroup, based on shared
characteristics
o in each age cohort the same X based on characteristics; gender, educational level
• Cluster sample: clinics units (30) are collaborating together and pick a couple clinical
units (4). And then all people of 4 units take in research.
o Challenge for clinical research: find clinics that want to collaborate to find
patients. And limited capacity so research study is mostly not a priority.
→ In real world we don’t randomly select people, and ‘force’ clinic units/people to participate
Measurement (accuracy, precision, scaling, sensitivity, computer vs paper-pencil)
• Accuracy (=validity) - based on Continuous Test Performance
o Discrimental validity: group with attentional problems (brain damage/ADHD) and
compare them to healthy control group. Lower scores for the group we expect to
have lower scores and so have attentional problems
o Incremental validity: it adds something → questionnaire with items about
attentional problems. The test is adding something to differentiate the groups to my
questionnaires?
o Validate something new, you need a gold standard to find out to make associations
and test to prove X (attention) deficits. You need consensus on what attention is.
Gold standard needed in order to validate new tests against the standards. Gold
standard should not be questionable. If so, than subjective selection of test that is
close to the one that you validate
o Be critical: do you really train attention (outcome measurement) or do you train
practice test performance
• Precision (=reliability): how reliable is measurement at a re-assessment
o Same position in your comparison group. So the best is the best, and the worst is the
worst. NOT look at the absolute score, but the relative position (=the rank based on
all the people) is the same.
Learning goals – understand:
- Basic study designs, such as cross-sectional studies, case control studies, and cohort studies.
- Randomized Controlled Trials (e.g., following CONSORT statement)
o Classic state of intervention to find out if the intervention is working or not
- Single case research designs: Design, application, and analysis.
- Ethical considerations and research ethics in clinical neuropsychology.
- Pitfalls in the analysis of quasi-experimental clinical studies.
o With patients it is difficult to conclude causal effect ‘this is caused by’
o Taking the group as they are: brain tumor is not manipulated
o Sometimes overstep the data with correlational data
- Evaluating treatment efficacy: Consequences of expected effect size.
o P-value is the starting point, but actually you could skip it. But due the expectations
from the research field, you report it Effect Size (ES) is more important
- Clinical decision making using diagnostic tests.
- Reevaluation of patients with neuropsychological impairments.
- Selection of intervention methods: Understanding meta-analyses.
Do not:
- Study calculations or statistical output.
- Study formulas → these are only used for understanding the concepts. You should be able to
explain how these constructs are determined, which can be done by referring to formulas.
You can also explain in words how certain terms are determined.
Course literature
- Book chapters.
- Research articles.
Exam
- 6 open questions on various topics of the course (each topic devoted to one questions which
also contains sub-questions)
- Maximum of 60 points.
- Answers in EN or NL
,Week 1 – Lecture 15 November – Introduction lecture
Introductory test
Effect Size (ES) = the size of the impairment
• Difficult to get significant effect in small groups, but ES can still conclude how large an effect
is, independently of the sample size (because ES is not rely on sample size)
• In large samples: small effects get significant
• ES helps to compare significant effects with each other
o Don’t compare if you only at the significance
Underpowered = Lower likelihood of finding a true existing effect. If there is an effect, you have too
few participants to find it. Assuming that there is an effect. Sample not big enough to show it and
miss true effect.
Active control group design = Control group also receives the treatment. If our treatment works for a
group, compared to a control group
• Dose control: same substance but lower dose in pharmacology therapy
• Dismantling design: complex treatment (a lot of steps in assessment/treatment). We do
exactly the same, we take out one active step (don’t give feedback on the results OR remove
cognitive training)
• Waitlist control group: do nothing but will promised to receive the treatment later
o Treatment group compared to nothing is not what want to achieve with treatment
results
Mild cognitive impairment (MCI). Which statistical information do you describe for the test validity?
• Overall diagnostic accuracy (AUC)
• Spectivity: towards no disease
• Sensitivity: towards disease
→ how often do I commit misclassification
Explore effectiveness of treatment on single patient/ single case studies
• ABAB design: give treatment and return to baseline
• Multiple baseline design: with single patients
Reliable Change Index: indicates the change in a measure from one time to another (before and
after treatment) ON A SINGLE PERSON. Taking into consideration the variability of scores at the first
time, learning effects, practice effects, as well as the test-retest reliability
• Current state is different than in first assessment
o Negative score: with progressive dementia it further declines 2nd assessment
compared to 1st assessment
Delphi Methodology = includes a group of experts on a topic (the expert panel) and facilitates
collective decision making by a structured and anonymous way of communication
→ Exam: not ask for definitions. But askes to give examples, (dis)advantages
,Group design
Same time point
• Cross Sectional: one time point right now (patient and control sample) and compare them
(patient and control sample)
• Case-control: for each patient, you select a control person (same age, gender, educational
level). So for each patient a single matching control person.
o But most time this is not happening on individual level, but on group level
More time points
• Longitudinal study: follow patients over time
• Prospective: now recruitment and see how develop control
• Retrospective: talk to school, friends, parents and look at time BEFORE assessment.
o Bit more difficult
Aim of research is real truth (in world) how patients with X (dementia) will perform in Y (prospective
memory).
Real truth is not possible, because we cannot take all dementia patients as participants. Therefore do
an assessment to come close to the truth, by selection that will represent the population.
External validity = Phenomena of interest on intended variables. ULTIMATE GOAL
• How people struggle and test in REAL life. Really important because we don’t do
experimental research. But we want to know how patients perform, learn and struggle in life.
Test result is approximately, and not the real situation.
• Variable is hopefully represent the phenomena of interests/target population
Internal validity = Intended sample on actual subjects.
, • When I assess my people, I make certain mistakes (confounding variables, computer is not
working, not understanding instructions, mistaking in time measurements) and may deviate
from plan.
• Experimental psychologists to take care for low mistakes as possible by protocols, lab
Selection and recruitment = to reach a represented true population in sample
• Inclusion: demographic characteristics, clinical characteristics, geopathic, temporal
• Exclusion: risk of begin lost at follow up, inability to provide good date, at risk possible
adverse effects, non-representative for population
→ Aim: to represent the population with the right characteristics
Sampling: how do I select people
• Non-probability samples
o Convenience samples: people who are accessible to me, interested to take
part, approaching the network. Most studies are convenience samples.
▪ Difficult to interpret findings and raw conclusions for general population.
Because it depends on collaboration of participants → interpretations
drawing with care
• Snowball sampling
• Probability sample (idea situation: random select, people willing to participate, have time,
motivation)
• Simple random sampling: randomly drawn from population
• Systematic random sample: not completely random, but in each age cohort random (20-
30, 30-40, 40-50 etc)
• Stratified random sample: divides population into smaller subgroup, based on shared
characteristics
o in each age cohort the same X based on characteristics; gender, educational level
• Cluster sample: clinics units (30) are collaborating together and pick a couple clinical
units (4). And then all people of 4 units take in research.
o Challenge for clinical research: find clinics that want to collaborate to find
patients. And limited capacity so research study is mostly not a priority.
→ In real world we don’t randomly select people, and ‘force’ clinic units/people to participate
Measurement (accuracy, precision, scaling, sensitivity, computer vs paper-pencil)
• Accuracy (=validity) - based on Continuous Test Performance
o Discrimental validity: group with attentional problems (brain damage/ADHD) and
compare them to healthy control group. Lower scores for the group we expect to
have lower scores and so have attentional problems
o Incremental validity: it adds something → questionnaire with items about
attentional problems. The test is adding something to differentiate the groups to my
questionnaires?
o Validate something new, you need a gold standard to find out to make associations
and test to prove X (attention) deficits. You need consensus on what attention is.
Gold standard needed in order to validate new tests against the standards. Gold
standard should not be questionable. If so, than subjective selection of test that is
close to the one that you validate
o Be critical: do you really train attention (outcome measurement) or do you train
practice test performance
• Precision (=reliability): how reliable is measurement at a re-assessment
o Same position in your comparison group. So the best is the best, and the worst is the
worst. NOT look at the absolute score, but the relative position (=the rank based on
all the people) is the same.
