Research Methods in Clinical
Neuropsychology
Lecture 1 Basic study designs
Learning goals
- Pitfalls in the analysis of quasi-experimental clinical studies
- Evaluating treatment efficacy: consequences of expected effect size
- Clinical decision making using diagnostic tests
- Reevaluation of patients with neuropsychological impairments
- Selection of intervention methods: understanding meta-analyses
Group designs: selection, recruitment, measurement
One time point (now), cross sectional patient population compared to
control population. Sometimes patient (and controls) are also followed
over time, this is called prospective longitudinal. Or start now and ask
patients (and controls) how they were doing before, this is called
retrospective longitudinal. Case control means that there is a control
person for each
patient.
,The principle: we are interested in the ‘truth in the universe’, we want to
know how a target population does in a phenomenon of interest (for
example brain injury patients and attention). To find out we do a study,
that gives a sample of the population. The phenomena aren’t measurable
(we can’t fully capture attention), but we have variables that represent the
phenomena which we can measure with tests, observations etc.
External validity, how are they doing in daily live?
In the sample we make mistakes, have biases, dropouts, mistakes in
measuring, incorrect scoring etc. Internal validity.
Selection and recruitment
Establishing selection criteria
Inclusion criteria: demographic characteristics, clinical characteristics,
geographic characteristics, temporal characteristics.
Exclusion criteria: risk of being lost at follow up, inability to provide good
data, at risk for possible adverse effects, non-representative for
population.
Sampling - Nonprobability samples: convenience samples, snowball
sampling
Probability samples: simple random samples, systematic random samples,
stratified random sample, cluster sample (but random sample is almost
impossible in clinical setting).
Measurement
What measure should be selected for a specific study. Think about:
scaling, sensitivity, accuracy (validity) and precision (reliability) and
computerized or paper-pencil assessment?
Scaling - Categorical or continuous variables? Continuous variables contain
more information, are more flexible and often preferred.
Precision (reliability) - Degree to which individuals retain relative position
within distribution of scores from one testing session to another. Most
often presented as correlation in test-retest reliability (stability). Change
can be either high or small, but reliability has to be high.
Accuracy (validity) - Does the test measure what it should measure?
Sensitivity - You need to know about the population you are testing.
Sensitive measure required that detects improvements even in healthy
people.
How to reduce random error - Standardizing the measurement, training
the staff doing assessment, automating the instrument, blinding,
repeating the measurement.
Modern assessment methods: computerized vs. paper-pencil assessment –
There are different reasons that may explain reluctance:
1. Psychometric obstacles: Reliability of traditional and computerized
tests; equivalence of computer tests and paper-and-pencil tests; quality of
normative data.
2. Technical obstacles: E.g. speed in technical developments hamper work
on psychometric properties
, 3. Theoretical obstacles: Extensiveness of the body of knowledge;
theoretical paradigms and their practical value
4. Strategic obstacles: Rapidly growing number of new approaches results
in incomparability of results.
Some ideas about the future of neuropsychological testing – Save time and
less effort in scoring.
Computerized adaptive testing: Gain in efficiency (time and
precision) by selecting next item based on performance on current item.
Nominal response model: Different meanings on different types of errors
(e.g. response style on TMT). Test linking: Direct comparison of tests and
integration into individual report (e.g. ANDI.nl). Person fit statistics:
Performance validity based on fit of one item relative to overall
performance. Web-based testing, mobile platforms, wearables:
Assessment goes out of clinic into “real life”, increase in longitudinal
repeated assessments.
Attention deficits after traumatic brain injury – Example
Clinical observation: decline in performance over time. Affecting various
activities of daily living. Hypothesis: attention deficit in patients with TBI?
Attention was measured with the Continuous Performance Test (CPT),
computerized assessment of selective and sustained attention. Driving
performance was measured with a driving stimulator ride (daily life
performance).
Pitfalls in significance testing
- Post-hoc testing
- Cherry picking
- Nonspecific hypotheses
- Multiple testing
- Correction for multiple testing
Post-hoc testing - A priori hypothesis I, based on data exploration you
draw conclusions. You need a different dataset to confirm.
Random assignment to experimental groups, no hypotheses of group
differences on demographic variables. No testing necessary.
Cherry picking - Beforehand set measurements. Cherry picking means
focusing on results that fit your idea.
Nonspecific hypothesis - Hypothesis: mental disorders have a genetic
underpinning. Correlation of 1000 gene loci with 10 mental disorders,
10000 possible specific hypotheses.
Do an alpha level adjustment. Stepwise hypotheses generating and testing
is better. Exploratory (hypothesis-generating) study: No significance
testing. Hypothesis testing on independent data set (on specific
hypothesis)
Neuropsychology
Lecture 1 Basic study designs
Learning goals
- Pitfalls in the analysis of quasi-experimental clinical studies
- Evaluating treatment efficacy: consequences of expected effect size
- Clinical decision making using diagnostic tests
- Reevaluation of patients with neuropsychological impairments
- Selection of intervention methods: understanding meta-analyses
Group designs: selection, recruitment, measurement
One time point (now), cross sectional patient population compared to
control population. Sometimes patient (and controls) are also followed
over time, this is called prospective longitudinal. Or start now and ask
patients (and controls) how they were doing before, this is called
retrospective longitudinal. Case control means that there is a control
person for each
patient.
,The principle: we are interested in the ‘truth in the universe’, we want to
know how a target population does in a phenomenon of interest (for
example brain injury patients and attention). To find out we do a study,
that gives a sample of the population. The phenomena aren’t measurable
(we can’t fully capture attention), but we have variables that represent the
phenomena which we can measure with tests, observations etc.
External validity, how are they doing in daily live?
In the sample we make mistakes, have biases, dropouts, mistakes in
measuring, incorrect scoring etc. Internal validity.
Selection and recruitment
Establishing selection criteria
Inclusion criteria: demographic characteristics, clinical characteristics,
geographic characteristics, temporal characteristics.
Exclusion criteria: risk of being lost at follow up, inability to provide good
data, at risk for possible adverse effects, non-representative for
population.
Sampling - Nonprobability samples: convenience samples, snowball
sampling
Probability samples: simple random samples, systematic random samples,
stratified random sample, cluster sample (but random sample is almost
impossible in clinical setting).
Measurement
What measure should be selected for a specific study. Think about:
scaling, sensitivity, accuracy (validity) and precision (reliability) and
computerized or paper-pencil assessment?
Scaling - Categorical or continuous variables? Continuous variables contain
more information, are more flexible and often preferred.
Precision (reliability) - Degree to which individuals retain relative position
within distribution of scores from one testing session to another. Most
often presented as correlation in test-retest reliability (stability). Change
can be either high or small, but reliability has to be high.
Accuracy (validity) - Does the test measure what it should measure?
Sensitivity - You need to know about the population you are testing.
Sensitive measure required that detects improvements even in healthy
people.
How to reduce random error - Standardizing the measurement, training
the staff doing assessment, automating the instrument, blinding,
repeating the measurement.
Modern assessment methods: computerized vs. paper-pencil assessment –
There are different reasons that may explain reluctance:
1. Psychometric obstacles: Reliability of traditional and computerized
tests; equivalence of computer tests and paper-and-pencil tests; quality of
normative data.
2. Technical obstacles: E.g. speed in technical developments hamper work
on psychometric properties
, 3. Theoretical obstacles: Extensiveness of the body of knowledge;
theoretical paradigms and their practical value
4. Strategic obstacles: Rapidly growing number of new approaches results
in incomparability of results.
Some ideas about the future of neuropsychological testing – Save time and
less effort in scoring.
Computerized adaptive testing: Gain in efficiency (time and
precision) by selecting next item based on performance on current item.
Nominal response model: Different meanings on different types of errors
(e.g. response style on TMT). Test linking: Direct comparison of tests and
integration into individual report (e.g. ANDI.nl). Person fit statistics:
Performance validity based on fit of one item relative to overall
performance. Web-based testing, mobile platforms, wearables:
Assessment goes out of clinic into “real life”, increase in longitudinal
repeated assessments.
Attention deficits after traumatic brain injury – Example
Clinical observation: decline in performance over time. Affecting various
activities of daily living. Hypothesis: attention deficit in patients with TBI?
Attention was measured with the Continuous Performance Test (CPT),
computerized assessment of selective and sustained attention. Driving
performance was measured with a driving stimulator ride (daily life
performance).
Pitfalls in significance testing
- Post-hoc testing
- Cherry picking
- Nonspecific hypotheses
- Multiple testing
- Correction for multiple testing
Post-hoc testing - A priori hypothesis I, based on data exploration you
draw conclusions. You need a different dataset to confirm.
Random assignment to experimental groups, no hypotheses of group
differences on demographic variables. No testing necessary.
Cherry picking - Beforehand set measurements. Cherry picking means
focusing on results that fit your idea.
Nonspecific hypothesis - Hypothesis: mental disorders have a genetic
underpinning. Correlation of 1000 gene loci with 10 mental disorders,
10000 possible specific hypotheses.
Do an alpha level adjustment. Stepwise hypotheses generating and testing
is better. Exploratory (hypothesis-generating) study: No significance
testing. Hypothesis testing on independent data set (on specific
hypothesis)
