Article 1: A Tutorial on Mechanical Decision-Making for Personnel and Educational Selection
Rob R. Meijer, Marvin Neumann, Bas T. Hemker and A. Susan M. Niessen
Abstract
In decision-making, it is important not only to use the correct information but also to combine information in an optimal
way. There are robust research findings that a mechanical combination of information for personnel and educational
selection matches or outperforms a holistic combination of information. However, practitioners and policy makers seldom
use mechanical combination for decision-making. One of the important conditions for scientific results to be used in practice
and to be part of policy-making is that results are easily accessible. To increase the accessibility of mechanical judgment
prediction procedures, we:
1. Explain in detail how mechanical combination procedures work,
2. Provide examples to illustrate these procedures
3. Discuss some limitations of mechanical decision-making.
Introduction
Decision-making is essential in recruitment, selection, and admissions, relying heavily on selecting appropriate predictors
and gathering relevant data. However, the quality of decisions depends not only on the information but also on how it is
combined. Empirical research indicates that combining information through decision rules yields better results than relying
on intuitive judgment (e.g., Meehl, 1954; Grove and Meehl, 1996). A well-known example is the Apgar score for newborns,
where scores are assigned on five dimensions instead of relying on subjective judgment.
Mechanical or statistical decision-making, which combines information using a set rule, outperforms holistic (intuitive)
decision-making in predicting outcomes, such as in hiring decisions. Despite this, holistic judgment remains widely used in
practice, such as in educational admissions, where factors like hardships or community service are often considered
subjectively. Researchers like Highhouse (2008) suggest that resistance to mechanical methods stems from a lack of
awareness of their benefits and a desire for autonomy and social interaction in decision-making processes.
The underutilization of mechanical decision-making illustrates broader challenges in adopting evidence-based practices.
Although the effectiveness of mechanical judgment is well-established (e.g., Grove et al., 2000; Kuncel et al., 2013), it is
often seen as irrelevant by practitioners, who do not recognize their decisions as suboptimal. To overcome this, it is crucial
to make research findings accessible and demonstrate the advantages of mechanical decision-making.
This paper aims to address these challenges by explaining mechanical judgment procedures, providing examples, and
discussing common objections and overlooked benefits. By clarifying these procedures, the paper seeks to promote a more
widespread understanding and adoption of mechanical decision-making, enhancing decision-making practices in selection
and recruitment.
Mechanical versus holistic judgement
While holistic judgment involves making decisions based on intuitive feelings and a personal understanding of the situation,
mechanical judgment combines information based on explicit rules or formulas, ensuring consistency. Studies consistently
show that mechanical judgment tends to be more predictive and accurate. For example, when comparing scores from
interviews and intelligence tests, combining the scores using a set rule gives more valid predictions than intuitively judging
the candidate's overall suitability.
An illustration: How to decide which candidate to select for a job
Step 1. Specify your criteria.
- What do you want to predict? For example, do you want to predict how well candidates will perform their
future tasks, or do you also want to select a friendly colleague?
Step 2. Choose your predictors.
- You decide what information to collect in order to make the desired predictions and how to collect that
- information for example, include scores on psychological tests and ratings based on structured interviews.
- It is important that the information that is considered is valid.
- A mechanical combination of information (predictors) does not imply that “subjective” impressions cannot be
considered: the combination rule is mechanical, not the information collection.
Step 3. Collect the information.
- Administering tests, conducting interviews, and/or rating resumes. The information is collected without making
judgments or decisions other than on the traits, skills, and abilities that are assessed.
Step 4. Combine information according to a rule.
- We will illustrate this below using a number of alternatives: (1) equal weighting, (2) weights obtained via experts,
(3) holistic and mechanical synthesis (Sawyer, 1966), and (4) limited expert judgment.
- One important condition for combining the information is using the same scale for all traits and skills that are
assessed This may be done through standardizing the scores and ratings.
- Finally, compute the final scores and decide who to select.
, Different ways to combine information
The most important advantage of using a rule is that it will result in consistent weights for the different types of information.
Method 1: Equal Weighting
- In this procedure, each predictor score is weighted equally and gets a weight of 1 Thus, each score or rating is
considered equally important.
- This simple rule can be applied for every candidate, and the candidate with the highest mean score across the
raters is selected.
- This method often works almost as well as regression-based weighting, especially when the sample on which
regression analysis is conducted is not large (say, not larger than 200) and when the optimal regression-based
weights would not differ much from each other
- However, when we apply equal weighting and (1) there is one strong predictor (such as intelligence test scores),
(2) the other predictors have a relatively weak relation with the criterion, and (3) the intercorrelations between
the predictors are relatively strong, adding a second or third predictor may reduce the predictive validity as
compared to using only one best predictor.
Method 2: Weights provided by experts
- Decision makers may have reasons to consider particular information about a candidate more important than
other information for example, we can decide to give more weight to the cognitive ability test score because
research has shown that these test scores have, in general, a strong relationship with future job performance.
Method 3: Mechanical synthesis
- Under this rule, a decision maker first makes a holistic judgment of the suitability of the candidate.
- In general, this procedure does not result in better decisions compared to the procedures described above, but it
also does not result in reduced accuracy, unless the holistic rating would receive a high weight.
- The advantage is that it increases the sense of control and autonomy of decision makers
Method 4: Holistic synthesis
- Here, we start using a rule such as discussed in method 1 or 2, but after the scores have been calculated
mechanically, the decision maker is allowed to combine the mechanical prediction with all other information
holistically and may change the final rating accordingly This enhances the feeling of autonomy.
- We hypothesize that this method probably yields more acceptance and higher use-intentions from decision
makers compared to the “purer” mechanical procedures described previously, but lower accuracy as compared to
pure mechanical rules without the possibility to overrule the results.
- Finally, when a large pool of candidates is available, Kuncel (2008) proposed to first select candidates
mechanically, and then select the final candidates holistically.
Commonly raised objectives and often overlooked advantages
More information is not always better
What is often overlooked is that more information does not always lead to better prediction. It is advised to
remove information that has small predictive validity and would be given a small weight from assessment
procedures altogether.
Reliability of scores and ratings
When making decisions, it is often inevitable that there will be persons with similar scores In those cases, considering
candidates whose confidence intervals overlap as similar is not an option because of the logical inconsistencies that follow
from this approach. Thus, how should we decide which candidate to select? The best option is still to pick the candidates
with the highest scores (possibly obtained across different predictors), and to ignore information about reliability or stability
of scores in the part of the process in which decisions are made based on the information that was collected.
Predicting multiple outcomes
In some cases, we may not want to predict one outcome, but multiple outcomes that are not strongly related to each other,
such as task performance and turnover, or diversity and adverse impact reduction. Another issue is the assumption that
higher scores and ratings translate to higher job performance. However, if moderate scores are desirable, it is possible to
give higher scores for moderate levels of a traits or skill in mechanical procedures, if that would be warranted.
Transparency
An important advantage of mechanical decision-making is that decision makers can be completely transparent about how
they reach a decision. Transparent procedures allow for evaluation and improvement, but they also make the flaws and
errors of judgments and decisions visible.
Limitations of mechanical judgements
Perhaps the biggest limitation in practice is that decision makers do not like to apply mechanical decision-making
because it weakens basic human needs such as autonomy, competence, and relatedness.
Rob R. Meijer, Marvin Neumann, Bas T. Hemker and A. Susan M. Niessen
Abstract
In decision-making, it is important not only to use the correct information but also to combine information in an optimal
way. There are robust research findings that a mechanical combination of information for personnel and educational
selection matches or outperforms a holistic combination of information. However, practitioners and policy makers seldom
use mechanical combination for decision-making. One of the important conditions for scientific results to be used in practice
and to be part of policy-making is that results are easily accessible. To increase the accessibility of mechanical judgment
prediction procedures, we:
1. Explain in detail how mechanical combination procedures work,
2. Provide examples to illustrate these procedures
3. Discuss some limitations of mechanical decision-making.
Introduction
Decision-making is essential in recruitment, selection, and admissions, relying heavily on selecting appropriate predictors
and gathering relevant data. However, the quality of decisions depends not only on the information but also on how it is
combined. Empirical research indicates that combining information through decision rules yields better results than relying
on intuitive judgment (e.g., Meehl, 1954; Grove and Meehl, 1996). A well-known example is the Apgar score for newborns,
where scores are assigned on five dimensions instead of relying on subjective judgment.
Mechanical or statistical decision-making, which combines information using a set rule, outperforms holistic (intuitive)
decision-making in predicting outcomes, such as in hiring decisions. Despite this, holistic judgment remains widely used in
practice, such as in educational admissions, where factors like hardships or community service are often considered
subjectively. Researchers like Highhouse (2008) suggest that resistance to mechanical methods stems from a lack of
awareness of their benefits and a desire for autonomy and social interaction in decision-making processes.
The underutilization of mechanical decision-making illustrates broader challenges in adopting evidence-based practices.
Although the effectiveness of mechanical judgment is well-established (e.g., Grove et al., 2000; Kuncel et al., 2013), it is
often seen as irrelevant by practitioners, who do not recognize their decisions as suboptimal. To overcome this, it is crucial
to make research findings accessible and demonstrate the advantages of mechanical decision-making.
This paper aims to address these challenges by explaining mechanical judgment procedures, providing examples, and
discussing common objections and overlooked benefits. By clarifying these procedures, the paper seeks to promote a more
widespread understanding and adoption of mechanical decision-making, enhancing decision-making practices in selection
and recruitment.
Mechanical versus holistic judgement
While holistic judgment involves making decisions based on intuitive feelings and a personal understanding of the situation,
mechanical judgment combines information based on explicit rules or formulas, ensuring consistency. Studies consistently
show that mechanical judgment tends to be more predictive and accurate. For example, when comparing scores from
interviews and intelligence tests, combining the scores using a set rule gives more valid predictions than intuitively judging
the candidate's overall suitability.
An illustration: How to decide which candidate to select for a job
Step 1. Specify your criteria.
- What do you want to predict? For example, do you want to predict how well candidates will perform their
future tasks, or do you also want to select a friendly colleague?
Step 2. Choose your predictors.
- You decide what information to collect in order to make the desired predictions and how to collect that
- information for example, include scores on psychological tests and ratings based on structured interviews.
- It is important that the information that is considered is valid.
- A mechanical combination of information (predictors) does not imply that “subjective” impressions cannot be
considered: the combination rule is mechanical, not the information collection.
Step 3. Collect the information.
- Administering tests, conducting interviews, and/or rating resumes. The information is collected without making
judgments or decisions other than on the traits, skills, and abilities that are assessed.
Step 4. Combine information according to a rule.
- We will illustrate this below using a number of alternatives: (1) equal weighting, (2) weights obtained via experts,
(3) holistic and mechanical synthesis (Sawyer, 1966), and (4) limited expert judgment.
- One important condition for combining the information is using the same scale for all traits and skills that are
assessed This may be done through standardizing the scores and ratings.
- Finally, compute the final scores and decide who to select.
, Different ways to combine information
The most important advantage of using a rule is that it will result in consistent weights for the different types of information.
Method 1: Equal Weighting
- In this procedure, each predictor score is weighted equally and gets a weight of 1 Thus, each score or rating is
considered equally important.
- This simple rule can be applied for every candidate, and the candidate with the highest mean score across the
raters is selected.
- This method often works almost as well as regression-based weighting, especially when the sample on which
regression analysis is conducted is not large (say, not larger than 200) and when the optimal regression-based
weights would not differ much from each other
- However, when we apply equal weighting and (1) there is one strong predictor (such as intelligence test scores),
(2) the other predictors have a relatively weak relation with the criterion, and (3) the intercorrelations between
the predictors are relatively strong, adding a second or third predictor may reduce the predictive validity as
compared to using only one best predictor.
Method 2: Weights provided by experts
- Decision makers may have reasons to consider particular information about a candidate more important than
other information for example, we can decide to give more weight to the cognitive ability test score because
research has shown that these test scores have, in general, a strong relationship with future job performance.
Method 3: Mechanical synthesis
- Under this rule, a decision maker first makes a holistic judgment of the suitability of the candidate.
- In general, this procedure does not result in better decisions compared to the procedures described above, but it
also does not result in reduced accuracy, unless the holistic rating would receive a high weight.
- The advantage is that it increases the sense of control and autonomy of decision makers
Method 4: Holistic synthesis
- Here, we start using a rule such as discussed in method 1 or 2, but after the scores have been calculated
mechanically, the decision maker is allowed to combine the mechanical prediction with all other information
holistically and may change the final rating accordingly This enhances the feeling of autonomy.
- We hypothesize that this method probably yields more acceptance and higher use-intentions from decision
makers compared to the “purer” mechanical procedures described previously, but lower accuracy as compared to
pure mechanical rules without the possibility to overrule the results.
- Finally, when a large pool of candidates is available, Kuncel (2008) proposed to first select candidates
mechanically, and then select the final candidates holistically.
Commonly raised objectives and often overlooked advantages
More information is not always better
What is often overlooked is that more information does not always lead to better prediction. It is advised to
remove information that has small predictive validity and would be given a small weight from assessment
procedures altogether.
Reliability of scores and ratings
When making decisions, it is often inevitable that there will be persons with similar scores In those cases, considering
candidates whose confidence intervals overlap as similar is not an option because of the logical inconsistencies that follow
from this approach. Thus, how should we decide which candidate to select? The best option is still to pick the candidates
with the highest scores (possibly obtained across different predictors), and to ignore information about reliability or stability
of scores in the part of the process in which decisions are made based on the information that was collected.
Predicting multiple outcomes
In some cases, we may not want to predict one outcome, but multiple outcomes that are not strongly related to each other,
such as task performance and turnover, or diversity and adverse impact reduction. Another issue is the assumption that
higher scores and ratings translate to higher job performance. However, if moderate scores are desirable, it is possible to
give higher scores for moderate levels of a traits or skill in mechanical procedures, if that would be warranted.
Transparency
An important advantage of mechanical decision-making is that decision makers can be completely transparent about how
they reach a decision. Transparent procedures allow for evaluation and improvement, but they also make the flaws and
errors of judgments and decisions visible.
Limitations of mechanical judgements
Perhaps the biggest limitation in practice is that decision makers do not like to apply mechanical decision-making
because it weakens basic human needs such as autonomy, competence, and relatedness.
