Missing articles theme 7 vignette 3 - 5
He, J., & Van de Vijver, F. (2012). Bias and equivalence in cross-cultural research
This paper deals with methodological aspects of cross-cultural research, focusing on two key concepts: bias
and equivalence. Bias refers to nuisance factors that jeopardize the validity of instruments applied in
different cultures. Equivalence refers to the level of comparability of scores across cultures. There are two
important characteristics of bias and equivalence:
- Bias does not refer to random errors but to systematic measurement anomalies that are expected to be
replicable if a study were to be repeated.
- Equivalence is a characteristic of cross-cultural comparisons and not an intrinsic property of
instruments (e.g., both kilometers and miles are adequate units to measure distances, and any lack of
equivalence issues arise only in the comparison of both).
The less bias, the more equivalence.
1. Instrument choice in cross-cultural studies
An important question to consider in the initial stages of a project involves the choice of instruments:
- Adoption: amounts to a close translation of an instrument in a target language. This is most
frequently chosen in empirical research, because it is simple to implement, cheap, has a high face
validity, and retains the opportunity to compare scores obtained with the instrument across all
translations.
o However, it can only be used when the items in the source and target language version have
an adequate coverage of the construct measure. So, this option is only available if the
construct and instrument features (e.g., instructions and items) are taken to be adequate in all
cultural groups involved.
- Adaptation: it usually amounts to a combination of a close translation of some stimuli and a change
of other stimuli when a close translation would be inadequate for linguistic, cultural, or psychometric
reasons. It has become the generic term to refer to the translation process of psychological
instruments. It has become more common to view this process as requiring more than linguistic
skills, such as knowledge of the target culture, so as to be able to evaluate the psychological
relevance of the instrument in the new context. Weakness: more expensive, and time consuming.
- Assembly: it involves the compilation of a new instrument. This remains if adopting or adapting an
instrument will not produce an instrument with a satisfactory linguistic, cultural, and psychometric
accuracy. An assembly maximizes the cultural suitability of an instrument, but it will preclude any
numerical comparisons of scores across cultures.
- Selection criteria:
o If the aim is to maximize the opportunities for statistical comparisons in a study, adoption is
the simplest choice.
o If the aim is to maximize the ecological validity of the instrument (e.g., to adequately
measure the construct in a target culture), an adaptation or assembly is preferable.
o If the number of culture-specific items is large, the comparability of the construct or of the
remaining items may be problematic and opportunities for cross-cultural comparisons are
limited.
2. Taxonomy of bias
Bias occurs when score differences on the indicators of a particular construct do not correspond to
differences in the underlying trait or ability. This means that whereas a response in one culture represents a
target construct (e.g., conscientiousness), responses in another country are due to other constructs (e.g.,
social desirability), or additional constructs (a combination of conscientiousness and social desirability).
There are three types of bias:
, Construct bias
The construct measured is not identical across cultures. It can occur when there is only a partial overlap in
definition of the construct across cultures, or when not all relevant behaviors associated with the construct
present are proper samples in each culture. For example, happiness has a different focus in Western
(personal achievements) and non-Western countries (interpersonal connectedness). The outcome of the
assessment should acknowledge the incompleteness of overlap of the construct.
Method bias
This is a generic term for nuisance factors that derive from the sampling, structural features of the
instrument, or administration processes.
Sample bias
Results from incomparability of samples due to cross-cultural variation in sample characteristics that have a
bearing on target measures, such as confounding cross-cultural differences in education levels when testing
intelligence, variations in urban or rural residency, or in affiliation to religious groups. Due to resources and
accessibility restraints, randomly-sampled culturally representative respondents are rarely accomplished.
Many cross-cultural studies use college students, assuming that they constitute matching samples. However,
this may be invalid; for example, college education quality and enrolment rates in developed and developing
countries differ significantly, which can introduce selection biases in the sampling process.
To minimize sample bias, some suggested that the sampling of cultures should be guided by research goals.
When participants are recruited using convenience sampling, the generalization of findings to their
population can be problematic; the distribution of the target variable is to guide optimal sampling. If the
strategy to find matched samples does not work, it may well be possible to control for factors that induce
sample bias (such as the measurement of educational quality when assessing intelligence, so that a statistical
correction for the confounding differences can be achieved).
Instrument bias
Involves problems deriving from instrument characteristics, such as stimulus familiarity (in cognitive and
educational tests) and response styles (in personality and attitude inventories). Cultures tend to have
different levels of familiarity with stimulus materials, response modes, or response procedures. Such cross-
cultural differences in background characteristics tend to influence the scores on target measures.
Malda, Van de Vijver, and Temane confirmed the influence of content familiarity in their study of a
cognitive test in two cultures in South Africa. They found that children from the White and Black culture
performed better when the version was designed for their own group. Another study found that Chinese
children outperformed Greek children on tasks of visual-spatial processing, which could be attributed to
Chinese children’s intensive visual-spatial practice involved in learning to write Chinese. To tackle biases
arising from stimulus familiarity, tests should be locally adapted.
Response styles
Refer to a systematic tendency to use certain categories of the answering scale on some basis other than the
target construct. Acquiescence, the tendency to agree rather than disagree to propositions in general, is one
of the most prevalent response styles. This is more frequently endorsed by people with low socioeconomic
status from collectivistic cultures. Evidence suggests that the number of Likert point in rating scales may
induce different levels of response styles. Different standardization methods using information of means and
standard deviations in individuals or cultures have been proposed to control for response styles. To
determine the extent to which scores are influenced by response styles, correlations can be computed
between the corrected scores based on within-individual or within-cultural standardization of the raw scores.
The GLOBE leadership project applied a new approach to detect response styles, in which standardized
scores were used to predict the raw scores in a regression analysis, and then the raw scores were compared
with the predicted scores in t-tests to identify cultural exhibiting substantial response styles. Another way
involves the computation of response style scores, such as computing the proportion of items that are
He, J., & Van de Vijver, F. (2012). Bias and equivalence in cross-cultural research
This paper deals with methodological aspects of cross-cultural research, focusing on two key concepts: bias
and equivalence. Bias refers to nuisance factors that jeopardize the validity of instruments applied in
different cultures. Equivalence refers to the level of comparability of scores across cultures. There are two
important characteristics of bias and equivalence:
- Bias does not refer to random errors but to systematic measurement anomalies that are expected to be
replicable if a study were to be repeated.
- Equivalence is a characteristic of cross-cultural comparisons and not an intrinsic property of
instruments (e.g., both kilometers and miles are adequate units to measure distances, and any lack of
equivalence issues arise only in the comparison of both).
The less bias, the more equivalence.
1. Instrument choice in cross-cultural studies
An important question to consider in the initial stages of a project involves the choice of instruments:
- Adoption: amounts to a close translation of an instrument in a target language. This is most
frequently chosen in empirical research, because it is simple to implement, cheap, has a high face
validity, and retains the opportunity to compare scores obtained with the instrument across all
translations.
o However, it can only be used when the items in the source and target language version have
an adequate coverage of the construct measure. So, this option is only available if the
construct and instrument features (e.g., instructions and items) are taken to be adequate in all
cultural groups involved.
- Adaptation: it usually amounts to a combination of a close translation of some stimuli and a change
of other stimuli when a close translation would be inadequate for linguistic, cultural, or psychometric
reasons. It has become the generic term to refer to the translation process of psychological
instruments. It has become more common to view this process as requiring more than linguistic
skills, such as knowledge of the target culture, so as to be able to evaluate the psychological
relevance of the instrument in the new context. Weakness: more expensive, and time consuming.
- Assembly: it involves the compilation of a new instrument. This remains if adopting or adapting an
instrument will not produce an instrument with a satisfactory linguistic, cultural, and psychometric
accuracy. An assembly maximizes the cultural suitability of an instrument, but it will preclude any
numerical comparisons of scores across cultures.
- Selection criteria:
o If the aim is to maximize the opportunities for statistical comparisons in a study, adoption is
the simplest choice.
o If the aim is to maximize the ecological validity of the instrument (e.g., to adequately
measure the construct in a target culture), an adaptation or assembly is preferable.
o If the number of culture-specific items is large, the comparability of the construct or of the
remaining items may be problematic and opportunities for cross-cultural comparisons are
limited.
2. Taxonomy of bias
Bias occurs when score differences on the indicators of a particular construct do not correspond to
differences in the underlying trait or ability. This means that whereas a response in one culture represents a
target construct (e.g., conscientiousness), responses in another country are due to other constructs (e.g.,
social desirability), or additional constructs (a combination of conscientiousness and social desirability).
There are three types of bias:
, Construct bias
The construct measured is not identical across cultures. It can occur when there is only a partial overlap in
definition of the construct across cultures, or when not all relevant behaviors associated with the construct
present are proper samples in each culture. For example, happiness has a different focus in Western
(personal achievements) and non-Western countries (interpersonal connectedness). The outcome of the
assessment should acknowledge the incompleteness of overlap of the construct.
Method bias
This is a generic term for nuisance factors that derive from the sampling, structural features of the
instrument, or administration processes.
Sample bias
Results from incomparability of samples due to cross-cultural variation in sample characteristics that have a
bearing on target measures, such as confounding cross-cultural differences in education levels when testing
intelligence, variations in urban or rural residency, or in affiliation to religious groups. Due to resources and
accessibility restraints, randomly-sampled culturally representative respondents are rarely accomplished.
Many cross-cultural studies use college students, assuming that they constitute matching samples. However,
this may be invalid; for example, college education quality and enrolment rates in developed and developing
countries differ significantly, which can introduce selection biases in the sampling process.
To minimize sample bias, some suggested that the sampling of cultures should be guided by research goals.
When participants are recruited using convenience sampling, the generalization of findings to their
population can be problematic; the distribution of the target variable is to guide optimal sampling. If the
strategy to find matched samples does not work, it may well be possible to control for factors that induce
sample bias (such as the measurement of educational quality when assessing intelligence, so that a statistical
correction for the confounding differences can be achieved).
Instrument bias
Involves problems deriving from instrument characteristics, such as stimulus familiarity (in cognitive and
educational tests) and response styles (in personality and attitude inventories). Cultures tend to have
different levels of familiarity with stimulus materials, response modes, or response procedures. Such cross-
cultural differences in background characteristics tend to influence the scores on target measures.
Malda, Van de Vijver, and Temane confirmed the influence of content familiarity in their study of a
cognitive test in two cultures in South Africa. They found that children from the White and Black culture
performed better when the version was designed for their own group. Another study found that Chinese
children outperformed Greek children on tasks of visual-spatial processing, which could be attributed to
Chinese children’s intensive visual-spatial practice involved in learning to write Chinese. To tackle biases
arising from stimulus familiarity, tests should be locally adapted.
Response styles
Refer to a systematic tendency to use certain categories of the answering scale on some basis other than the
target construct. Acquiescence, the tendency to agree rather than disagree to propositions in general, is one
of the most prevalent response styles. This is more frequently endorsed by people with low socioeconomic
status from collectivistic cultures. Evidence suggests that the number of Likert point in rating scales may
induce different levels of response styles. Different standardization methods using information of means and
standard deviations in individuals or cultures have been proposed to control for response styles. To
determine the extent to which scores are influenced by response styles, correlations can be computed
between the corrected scores based on within-individual or within-cultural standardization of the raw scores.
The GLOBE leadership project applied a new approach to detect response styles, in which standardized
scores were used to predict the raw scores in a regression analysis, and then the raw scores were compared
with the predicted scores in t-tests to identify cultural exhibiting substantial response styles. Another way
involves the computation of response style scores, such as computing the proportion of items that are
