Methods of Assessment
Short Summary
Period 2
Maastricht University
Emma Leibbrand
Table of content
Task 2 – What does My Outcome Measure Reflect?
Task 3 – Neuropsychological Testing in Psychiatric Disorders
Task 4 – COVID-19: Screening the Aftermath
Task 5 – A Good Score Today, Poor Tomorrow
Task 6 – Additional Tools for NP Testing: Necessary for Future Diagnosing?
Task 7 – Clinical Neuropsychological Assessment in the 21 st Century
Task 8 – Ethical Issues of NP testing?
0
, [ Task 2 – What Does My Outcome Measure
Reflect? ]
(1) HOWIESON: CURRENT LIMITATIONS OF NEUROPSYCHOLOGICAL TESTS AND
ASSESSMENT PROCEDURES
This paper focuses on the limitations of current neuropsychological tests.
Existing tests and techniques can be improved; normative data for some neuropsychological tests are
based on small samples or have limited availability or reliability data.
Limitations of current neuropsychological tests:
o Many tests take too long to administer (e.g. Wisconsin Card Sorting test; 30 min.).
o Standard examination often minimises emotional influences, but important information can be
lost under such circumstances.
Tests exploring individual differences in emotional regulation should help identify
patients who are vulnerable to disruption of cognitive processes.
o The application of tests for diverse languages and cultures needs to be improved.
o The reliability and validity needs to be improved for some NP tests.
o Ecological validity needs to be improved => standard procedures provide advantages but they
also have limitations; standard procedures often deviate from real-world demands.
o Some tests have become too familiar through internet searches and publicity.
o Many current tests are paper-and-pencil tests, but NP assessment is on the verge of some
changes in procedures, including computerised tests.
Advantage => measuring speed and response consistency.
Disadvantage => some aspects of performance not captured (e.g. free recall).
Foundation of examination procedures
Modern roots of clinical neuropsychological assessment extend to the mid-twentieth century and can be
traced largely to the work of Luria, who developed a theory of brain-behaviour relationships and
instruments for studying higher cortical functions from his study of WW2 soldiers that sustained gunshots
wounds to the brain.
During the following decades many people with brain lesions were studied => fuller understanding.
We are learning that assumptions about what brain areas are responsible for particular cognitive tasks
based on lesion studies are wrong or incomplete in some cases.
o Advances in neuroimaging made it possible to image networks of brain activity.
o Example => WCST interpreted as example of EF based on the requirement that success depends
on WM, concept attainment, maintaining behaviour based on rewards, and shifting behaviour
when rewards shift to a new concept, all generally regarded as frontal lobe functions.
Limitation of fMRI are => rather a research than clinical tool + standard NP tests not suitable for fMRI
because they are too lengthy or require an action by patient that requires the head to move.
Psycho-legal issues
Neuropsychological evaluations are significantly important in cases related to disabilities, decision-making
capacities, civil and criminal competencies, and responsibilities.
Forensic NP evaluations require a high level of expertise => not all practitioners in this field possess it.
Cultural factors are vital when interpreting data in forensic evaluations.
Normative data often does not represent justice-involved individuals due to unique demographic and
clinical factors.
Assessing response bias is challenging => variability among clinicians.
Better understanding of what is meant by invalid performance is needed in these evaluations.
1
,(2) TUOKKO: NEUROPSYCHOLOGICAL DETECTION OF COGNITIVE IMPAIRMENT
This paper examines the agreement between neuropsychologists identifying cognitive impairment (CI) in
older adults + the factors influencing the classification process.
o Identification of CI is critically important to the health and well-being of older adults.
CI is a heterogeneous classification and its identification relies heavily on a broad understanding of brain-
behaviour relations across the lifespan.
o Many different underlying disorders may result in impaired cognitive functioning
Given the complexity and importance of this task, it is vital to ascertain whether clinicians draw similar
conclusions when provided with the same information.
o That is, the inter-agreement when identifying cognitive impairment is one of the utmost
importance for neuropsychologists.
o Both the characteristics of the participants (e.g., sensory impairment, health status) and the
raters (e.g., experience, training, confidence levels) may influence clinical decision-making.
This study aims to assess inter-related agreement of neuropsychologists in classifying cognitive impairment
and identify factors influencing their decision-making process.
o It anticipated two types of concordant cases: clearly not impaired and clearly impaired
individuals. Discordant cases are expected to fall between these two groups in measures of
cognitive functioning.
o Concordant => clinicians agree; discordant => clinicians do not agree.
o Characteristics like age, education, gender, cognitive scores, depression, and premorbid IQ
may differentiate these groups.
o Rater characteristics may also influence the decision-making process due to the homogeneity
of the study population and the shared information.
In subtle or mild disorders, it is harder to achieve inter-agreement than when a disorder is really severe.
When clinicians rate a patient on level of impairment, they have to send it to a second neuropsychologists
who then has to follow the same procedure.
o The kappa was .49 falls within the moderate agreement => two raters share a fair amount of
agreement but there is room for improvement.
Results
Neuropsychologists showed a moderate level of agreement when asked to judge whether a participant
exhibited CI or not.
Kappa was .49, which is similar to existing literature for psychologists and neurologists making similar
judgements.
Well-defined criteria + standardised vignettes or multidisciplinary approach => higher inter-agreement.
The only significant difference between concordant and discordant groups was cognitive performance,
with discordant cases falling between the clearly impaired and not impaired groups.
Half of the discordant cases => consensus diagnosis of Cognitive Impairment No Dementia (CIND),
(potentially in the early stages of progressive dementia).
Disagreements among neuropsychologists primarily occur in cases with subtle or mild CI.
o There is ongoing controversy in defining mild cognitive impairment.
o Achieving better inter-related agreement depends on reaching a consensus on the
classification of mild CI and defining it more clearly.
(3) CHAYTOR: IMPROVING THE ECOLOGICAL VALIDITY OF EF ASSESSMENT
This paper investigates ways to improve the ecological validity of NP assessment of EF
o Done through compensatory strategies and environmental cognitive demands.
Neuropsychologists often perplexed by inconsistencies between clients’ performance on EF tests and
everyday executive functioning abilities.
o It is assumed that the impaired brain processes, which lead to poor performance on a test,
will also lead to poor performance in other situations.
o In other words, it is assumed that neuropsychological tests have ecological validity.
There has been little research examining the ecological validity of EF assessment.
2
, o Important to research on this => how everyday cognitive ability is measured.
o Previous research implemented self and informant questionnaires, clinician rating scales, or
observation of simulated everyday tasks.
The literature investigating the ecological validity of NP tests of EF has been inconsistent: some studies
demonstrating robust relationships between test scores and everyday ability, and other failing to find this.
o It remains difficult to compare findings across studies as they evaluated different populations
(e.g. schizophrenia, ADHD, older adults, etc.), the type and number of NP tests of EF, and the
outcome measure utilised.
Despite these differences, even in the studies that reported ecological validity, the magnitude tends to be
moderate => large amount of the variance in everyday EF skills remains unaccounted for.
Many characteristics of traditional NP assessment can pose problems for ecological validity:
o The nature of the testing environment.
o Incomplete agreement on what construct the tests measure.
o The small sample of behaviour observed during testing.
o The inability to use compensatory strategies.
Non cognitive factors that influence the relationship between test performance and everyday performance
are => emotional problems, level of premorbid functioning, motor functioning, health problems, varying
environmental cognitive demands.
This study investigates the contribution of environmental cognitive demands and compensatory strategies.
Environmental => can vary widely across individuals + may mediate relationship between test
cognitive demands performance and everyday functioning.
Someone with EF deficit may have no real world problems if there is little
environmental demand.
Someone with minor EF deficits may have extreme functional impairment in a
highly demanding environment.
This highlights the need to assess the cognitive demands required of any person
to function in the person’s everyday environment + match these demands to
cognitive test performance to accurately predict functional consequences.
To date => very little attention to develop objective methods for assessing
environmental cognitive demands.
Currently => subjective approaches are used, such as clinician ratings of the
environment as having high/low demand in a given cognitive domain.
Inclusion of environmental demand assessment in studies of ecological validity
has the potential to clarify the relationship between NP tests and everyday
cognitive performance.
Compensatory => may also obscure relationship betw. test performance and everyday performance.
strategies
Someone may use compensatory skills in everyday functioning, but is prevented
from using them in the testing situation.
Someone may fail to use cognitive strategies in everyday situation where its
expected or needed.
Assessment of the individual’s everyday compensatory strategy use may add
important information when predicting every EF skills.
Results
Assessment of non-traditional variables, e.g. compensatory strategy use + environmental cognitive
demands, can account for additional variance in ratings of everyday EF skills.
o Accounting for these variables can improve relationships between EF tests and ratings of
everyday EF ability.
The study found that the ecological validity of tests might vary depending on the population being studied,
suggesting that different groups might respond differently to these tests.
3
Short Summary
Period 2
Maastricht University
Emma Leibbrand
Table of content
Task 2 – What does My Outcome Measure Reflect?
Task 3 – Neuropsychological Testing in Psychiatric Disorders
Task 4 – COVID-19: Screening the Aftermath
Task 5 – A Good Score Today, Poor Tomorrow
Task 6 – Additional Tools for NP Testing: Necessary for Future Diagnosing?
Task 7 – Clinical Neuropsychological Assessment in the 21 st Century
Task 8 – Ethical Issues of NP testing?
0
, [ Task 2 – What Does My Outcome Measure
Reflect? ]
(1) HOWIESON: CURRENT LIMITATIONS OF NEUROPSYCHOLOGICAL TESTS AND
ASSESSMENT PROCEDURES
This paper focuses on the limitations of current neuropsychological tests.
Existing tests and techniques can be improved; normative data for some neuropsychological tests are
based on small samples or have limited availability or reliability data.
Limitations of current neuropsychological tests:
o Many tests take too long to administer (e.g. Wisconsin Card Sorting test; 30 min.).
o Standard examination often minimises emotional influences, but important information can be
lost under such circumstances.
Tests exploring individual differences in emotional regulation should help identify
patients who are vulnerable to disruption of cognitive processes.
o The application of tests for diverse languages and cultures needs to be improved.
o The reliability and validity needs to be improved for some NP tests.
o Ecological validity needs to be improved => standard procedures provide advantages but they
also have limitations; standard procedures often deviate from real-world demands.
o Some tests have become too familiar through internet searches and publicity.
o Many current tests are paper-and-pencil tests, but NP assessment is on the verge of some
changes in procedures, including computerised tests.
Advantage => measuring speed and response consistency.
Disadvantage => some aspects of performance not captured (e.g. free recall).
Foundation of examination procedures
Modern roots of clinical neuropsychological assessment extend to the mid-twentieth century and can be
traced largely to the work of Luria, who developed a theory of brain-behaviour relationships and
instruments for studying higher cortical functions from his study of WW2 soldiers that sustained gunshots
wounds to the brain.
During the following decades many people with brain lesions were studied => fuller understanding.
We are learning that assumptions about what brain areas are responsible for particular cognitive tasks
based on lesion studies are wrong or incomplete in some cases.
o Advances in neuroimaging made it possible to image networks of brain activity.
o Example => WCST interpreted as example of EF based on the requirement that success depends
on WM, concept attainment, maintaining behaviour based on rewards, and shifting behaviour
when rewards shift to a new concept, all generally regarded as frontal lobe functions.
Limitation of fMRI are => rather a research than clinical tool + standard NP tests not suitable for fMRI
because they are too lengthy or require an action by patient that requires the head to move.
Psycho-legal issues
Neuropsychological evaluations are significantly important in cases related to disabilities, decision-making
capacities, civil and criminal competencies, and responsibilities.
Forensic NP evaluations require a high level of expertise => not all practitioners in this field possess it.
Cultural factors are vital when interpreting data in forensic evaluations.
Normative data often does not represent justice-involved individuals due to unique demographic and
clinical factors.
Assessing response bias is challenging => variability among clinicians.
Better understanding of what is meant by invalid performance is needed in these evaluations.
1
,(2) TUOKKO: NEUROPSYCHOLOGICAL DETECTION OF COGNITIVE IMPAIRMENT
This paper examines the agreement between neuropsychologists identifying cognitive impairment (CI) in
older adults + the factors influencing the classification process.
o Identification of CI is critically important to the health and well-being of older adults.
CI is a heterogeneous classification and its identification relies heavily on a broad understanding of brain-
behaviour relations across the lifespan.
o Many different underlying disorders may result in impaired cognitive functioning
Given the complexity and importance of this task, it is vital to ascertain whether clinicians draw similar
conclusions when provided with the same information.
o That is, the inter-agreement when identifying cognitive impairment is one of the utmost
importance for neuropsychologists.
o Both the characteristics of the participants (e.g., sensory impairment, health status) and the
raters (e.g., experience, training, confidence levels) may influence clinical decision-making.
This study aims to assess inter-related agreement of neuropsychologists in classifying cognitive impairment
and identify factors influencing their decision-making process.
o It anticipated two types of concordant cases: clearly not impaired and clearly impaired
individuals. Discordant cases are expected to fall between these two groups in measures of
cognitive functioning.
o Concordant => clinicians agree; discordant => clinicians do not agree.
o Characteristics like age, education, gender, cognitive scores, depression, and premorbid IQ
may differentiate these groups.
o Rater characteristics may also influence the decision-making process due to the homogeneity
of the study population and the shared information.
In subtle or mild disorders, it is harder to achieve inter-agreement than when a disorder is really severe.
When clinicians rate a patient on level of impairment, they have to send it to a second neuropsychologists
who then has to follow the same procedure.
o The kappa was .49 falls within the moderate agreement => two raters share a fair amount of
agreement but there is room for improvement.
Results
Neuropsychologists showed a moderate level of agreement when asked to judge whether a participant
exhibited CI or not.
Kappa was .49, which is similar to existing literature for psychologists and neurologists making similar
judgements.
Well-defined criteria + standardised vignettes or multidisciplinary approach => higher inter-agreement.
The only significant difference between concordant and discordant groups was cognitive performance,
with discordant cases falling between the clearly impaired and not impaired groups.
Half of the discordant cases => consensus diagnosis of Cognitive Impairment No Dementia (CIND),
(potentially in the early stages of progressive dementia).
Disagreements among neuropsychologists primarily occur in cases with subtle or mild CI.
o There is ongoing controversy in defining mild cognitive impairment.
o Achieving better inter-related agreement depends on reaching a consensus on the
classification of mild CI and defining it more clearly.
(3) CHAYTOR: IMPROVING THE ECOLOGICAL VALIDITY OF EF ASSESSMENT
This paper investigates ways to improve the ecological validity of NP assessment of EF
o Done through compensatory strategies and environmental cognitive demands.
Neuropsychologists often perplexed by inconsistencies between clients’ performance on EF tests and
everyday executive functioning abilities.
o It is assumed that the impaired brain processes, which lead to poor performance on a test,
will also lead to poor performance in other situations.
o In other words, it is assumed that neuropsychological tests have ecological validity.
There has been little research examining the ecological validity of EF assessment.
2
, o Important to research on this => how everyday cognitive ability is measured.
o Previous research implemented self and informant questionnaires, clinician rating scales, or
observation of simulated everyday tasks.
The literature investigating the ecological validity of NP tests of EF has been inconsistent: some studies
demonstrating robust relationships between test scores and everyday ability, and other failing to find this.
o It remains difficult to compare findings across studies as they evaluated different populations
(e.g. schizophrenia, ADHD, older adults, etc.), the type and number of NP tests of EF, and the
outcome measure utilised.
Despite these differences, even in the studies that reported ecological validity, the magnitude tends to be
moderate => large amount of the variance in everyday EF skills remains unaccounted for.
Many characteristics of traditional NP assessment can pose problems for ecological validity:
o The nature of the testing environment.
o Incomplete agreement on what construct the tests measure.
o The small sample of behaviour observed during testing.
o The inability to use compensatory strategies.
Non cognitive factors that influence the relationship between test performance and everyday performance
are => emotional problems, level of premorbid functioning, motor functioning, health problems, varying
environmental cognitive demands.
This study investigates the contribution of environmental cognitive demands and compensatory strategies.
Environmental => can vary widely across individuals + may mediate relationship between test
cognitive demands performance and everyday functioning.
Someone with EF deficit may have no real world problems if there is little
environmental demand.
Someone with minor EF deficits may have extreme functional impairment in a
highly demanding environment.
This highlights the need to assess the cognitive demands required of any person
to function in the person’s everyday environment + match these demands to
cognitive test performance to accurately predict functional consequences.
To date => very little attention to develop objective methods for assessing
environmental cognitive demands.
Currently => subjective approaches are used, such as clinician ratings of the
environment as having high/low demand in a given cognitive domain.
Inclusion of environmental demand assessment in studies of ecological validity
has the potential to clarify the relationship between NP tests and everyday
cognitive performance.
Compensatory => may also obscure relationship betw. test performance and everyday performance.
strategies
Someone may use compensatory skills in everyday functioning, but is prevented
from using them in the testing situation.
Someone may fail to use cognitive strategies in everyday situation where its
expected or needed.
Assessment of the individual’s everyday compensatory strategy use may add
important information when predicting every EF skills.
Results
Assessment of non-traditional variables, e.g. compensatory strategy use + environmental cognitive
demands, can account for additional variance in ratings of everyday EF skills.
o Accounting for these variables can improve relationships between EF tests and ratings of
everyday EF ability.
The study found that the ecological validity of tests might vary depending on the population being studied,
suggesting that different groups might respond differently to these tests.
3
