Lecture 3 – Notes
Talent Assessment
Basic idea.
◦ Predicting future performance: what is/are the best predictor(s) X for criterion Y (y=
performance we ant to predict)
◦ but also about societal issues (e.g. how gets accepted to university)
Two major topics
1. clinical vs actuarial prediction → how do you combine information to make decisions
and predictions?
2. Signs & samples → what kind of information should we collect?
Clinical vs Actuarial Prediction
Do we need formal decision-making procedures?
◦ To hire suitable personnel
◦ to determine strengths and weaknesses
◦ to describe relevant behavior and traits
→ Can’t psychologists/expert decide based on their expertise/experience/intuition?
→ expert judgment vs decision rules!
How can you combine information (e.g. about candidate for job) to make a prediction?
Expert
judgment:
think they
get holistic
decision
Decision
rule: either
just add
information
up (all
weighted
the same)
or select
weights for
different
characteristics
Decision rule: transparent because can tell person how you cam up with decision & can
evaluate decision and check for improvements (not possible for expert judgment because
weighting implicit so do not know what to change)
What is the best method to predict outcomes?
Experiment: decision rule vs expert judgment
10 files of students applying for Psychology
standardized (test) info and responses from motivation statements
age, gender, nationality, previous education, admission test scores, answers to 2 motivational
questions (high school grades: 1-10, test score: 1-100)
for each student predict if they will pass the first years BSA norm (45 out of 60 credits) or
not
, this is a competition! (with a statistical formula derived from student data, but you have a lot
more information)
→ actual outcomes were 77% correct with decision rule (only considered math and overall score)
Formulas beat human judgment
we have known since 1954 (Paul Meehl)
the Golden Rule of Predictive Modeling:
◦ when based on the same evidence, the predictions of actuarial methods are at least as
valid, and are typically more valid, than the predictions of human experts
not only education: electroshock therapy, criminal recidivism etc
Even when human experts have more information than the formula, a formula performs
equal or better (but depends on quality of information → valid predictors)
Why?
Formula weighing information (optimally, equally/consistent) which people are not able to
do (no optimal weights: e.g. put to much weight on motivational factors and not enough on
test score; inconsistent: e.g. one time focused more on motivation, other time on scores)
a “model of man” is a better predictor than man itself (Dawes): even make statistical model
of how humans judges make judgments (e.g. how you weight specific information) that
model (of the judge) would lead to more accurate than the actual decision judge made
because the statistical model is consistent (apply same weights every time) while judge
himself is inconsistent
→ consistency in weights is more important than giving the optimal weight to information
What happens in practice?
Mostly clinical judgment is used
◦ we think we can process information better than a formula can
◦ see uniqueness (e.g. people say that might be on average but not in my particular case),
context and complex interactions (→ people say they would see complex interactions
which matter for decision making)
→ resistance to actuarial prediction! Some arguments:
You should use better judges → maybe some people are poor judges (but not me)
demanding perfect prediction by formulas
◦ actuarial predictions have low validity → however, no body says that about validity of
human judgment which is even worse! (actuarial predictions might not be perfect but sill
better than clinical judgments → explain at least a bit more variance and should be used
since there is no better alternative) Dawes
“I know someone who..” → people often come up with examples where rules were wrong
◦ cognitive conceit (assessor) → come up with examples where judgment based on human
judgment you were right and decision made on decision rules were wrong
◦ overconfidence in human judgments (assesses) → same as for assessor (e.g. someone
perform really bad at test but did very well on work)
→ do not like to be judged on numbers
actuarial prediction is dehumanizing
◦ is choosing to make poorer judgment than you can ethical?
Example – Cito test
◦ test at the end of primary school (age 12)
▪ academic achievement language and math
▪ largely determined secondary school level
◦ since 2015 (test only as second opinion
▪ largely determined by teachers judgment
▪ “the decision should be the teachers, who knows the child best”
▪ teacher judgment is clinical
Talent Assessment
Basic idea.
◦ Predicting future performance: what is/are the best predictor(s) X for criterion Y (y=
performance we ant to predict)
◦ but also about societal issues (e.g. how gets accepted to university)
Two major topics
1. clinical vs actuarial prediction → how do you combine information to make decisions
and predictions?
2. Signs & samples → what kind of information should we collect?
Clinical vs Actuarial Prediction
Do we need formal decision-making procedures?
◦ To hire suitable personnel
◦ to determine strengths and weaknesses
◦ to describe relevant behavior and traits
→ Can’t psychologists/expert decide based on their expertise/experience/intuition?
→ expert judgment vs decision rules!
How can you combine information (e.g. about candidate for job) to make a prediction?
Expert
judgment:
think they
get holistic
decision
Decision
rule: either
just add
information
up (all
weighted
the same)
or select
weights for
different
characteristics
Decision rule: transparent because can tell person how you cam up with decision & can
evaluate decision and check for improvements (not possible for expert judgment because
weighting implicit so do not know what to change)
What is the best method to predict outcomes?
Experiment: decision rule vs expert judgment
10 files of students applying for Psychology
standardized (test) info and responses from motivation statements
age, gender, nationality, previous education, admission test scores, answers to 2 motivational
questions (high school grades: 1-10, test score: 1-100)
for each student predict if they will pass the first years BSA norm (45 out of 60 credits) or
not
, this is a competition! (with a statistical formula derived from student data, but you have a lot
more information)
→ actual outcomes were 77% correct with decision rule (only considered math and overall score)
Formulas beat human judgment
we have known since 1954 (Paul Meehl)
the Golden Rule of Predictive Modeling:
◦ when based on the same evidence, the predictions of actuarial methods are at least as
valid, and are typically more valid, than the predictions of human experts
not only education: electroshock therapy, criminal recidivism etc
Even when human experts have more information than the formula, a formula performs
equal or better (but depends on quality of information → valid predictors)
Why?
Formula weighing information (optimally, equally/consistent) which people are not able to
do (no optimal weights: e.g. put to much weight on motivational factors and not enough on
test score; inconsistent: e.g. one time focused more on motivation, other time on scores)
a “model of man” is a better predictor than man itself (Dawes): even make statistical model
of how humans judges make judgments (e.g. how you weight specific information) that
model (of the judge) would lead to more accurate than the actual decision judge made
because the statistical model is consistent (apply same weights every time) while judge
himself is inconsistent
→ consistency in weights is more important than giving the optimal weight to information
What happens in practice?
Mostly clinical judgment is used
◦ we think we can process information better than a formula can
◦ see uniqueness (e.g. people say that might be on average but not in my particular case),
context and complex interactions (→ people say they would see complex interactions
which matter for decision making)
→ resistance to actuarial prediction! Some arguments:
You should use better judges → maybe some people are poor judges (but not me)
demanding perfect prediction by formulas
◦ actuarial predictions have low validity → however, no body says that about validity of
human judgment which is even worse! (actuarial predictions might not be perfect but sill
better than clinical judgments → explain at least a bit more variance and should be used
since there is no better alternative) Dawes
“I know someone who..” → people often come up with examples where rules were wrong
◦ cognitive conceit (assessor) → come up with examples where judgment based on human
judgment you were right and decision made on decision rules were wrong
◦ overconfidence in human judgments (assesses) → same as for assessor (e.g. someone
perform really bad at test but did very well on work)
→ do not like to be judged on numbers
actuarial prediction is dehumanizing
◦ is choosing to make poorer judgment than you can ethical?
Example – Cito test
◦ test at the end of primary school (age 12)
▪ academic achievement language and math
▪ largely determined secondary school level
◦ since 2015 (test only as second opinion
▪ largely determined by teachers judgment
▪ “the decision should be the teachers, who knows the child best”
▪ teacher judgment is clinical
