Edukacja 2014, 5(130), 92–111
An interdisciplinary approach
ISSN 0239-6858
Comparison of Mantel–Haenszel with IRT
procedures for DIF detection and effect size
estimation for dichotomous items
Bartosz Kondratek, Magdalena Grudniewska
Student Performance Analysis Unit, Educational Research Institute*
he article compares two methods used to detect diferential item functioning (DIF) of dichotomously scored
items: a nonparametric solution based on the Mantel–Haenszel procedure (MH) and a parametric IRT ap-
proach with a likelihood ratio test. A Monte Carlo experiment was performed in order to evaluate performance
of both statistics in various conditions of DIF uniformity. Results conirmed the theoretical prediction that the
MH test has greater statistical power in detecting uniform DIF than the likelihood ratio test and less power than
the LR test in cases of non-uniform DIF. Apart of examining statistical power of the test, speciic measures of
DIF efect size were compared: MH D–DIF and three measures of P–DIF expressed on the item easiness scale.
Keywords: diferential item functioning, Mantel–Haenszel test, item response theory.
D ifferential item functioning (DIF) is a sta-
tistical term that describes a condition
when the item response is dependent not only
G be the group membership variable, then
the general equation defining DIF with re-
gard to group membership will be of the form
on the ability level of the subject but also on the (c.f. Penfield and Camilli, 2007):
value of some additional group membership
variable. If DIF is present verification plays an which states that conditional distribution of
important role in psychometric analysis of a test the response is not explained solely by the
and is closely related to the problem of validity. ability variable (θ) of the examinee but is
Let Ui denote response to item i, θ be the additionally related to which group (G) the
level of ability that the test measures and examinee belongs to. In the case of dichoto-
mously scored item it can be rewritten as:
Article based on research carried out within the systemic which means that the probability of correct
project “Quality and effectiveness of education – strength-
ening of institutional research capabilities” executed by the response to the item is dependent not only
Educational Research Institute and co-financed from the on the ability θ, but also on the group mem-
European Social Fund (Human Capital Operational Pro- bership G. If G is two valued, G ∈ {f, r}, then
gramme 2007–2013, Priority III High quality of the edu- differential item functioning of item i can be
cation system). A preliminary version of this article was
also expressed as:
published primarily in Polish in Edukacja, 122(2) 2013.
*
Address: Zespół Analiz Osiągnięć Uczniów, Instytut (1)
Badań Edukacyjnych, ul. Górczewska 8, 01-180 Warszawa, meaning that the probability of correct re-
Poland. Email: sponse of an examinee with ability θ in group
, Comparison of Mantel–Haenszel with IRT procedures for DIF detection 93
uniform DIF non-uniform DIF non-uniform DIF
1
1
1
.8
.8
.8
P(U_i=1|theta)
P(U_i=1|theta)
P(U_i=1|theta)
.6
.6
.6
.4
.4
.4
.2
.2
.2
0
0
0
-4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4
theta theta theta
Figure 1. Examples of DIF (coninuous line depicts for: G = r, dashed line for: G = f).
f differs from the probability of correct re- The term item bias refers to the situation
sponse of an examinee with the same level of when one group is being favoured over an-
ability in group r. other group as a consequence of item con-
Figure 1 collects three examples of differ- tent unrelated to the ability intended to be
ential item functioning as defined by eq. (1) measured by the test. Item bias is a specific
by means of curves that depict the probabili- distortion of validity of the test and is not
ty of correct response conditional on θ (item equivalent to the presence of DIF. Differential
characteristic curve, ICC). The left-hand item functioning states that the item response
graph shows the uniform DIF – the ICC in is related to some additional factor that at the
one group is shifted parallel to the ICC for same time is unrelated to the ability mea-
the same item in the other group. All other sured by the test as a whole but is correlated
cases of DIF will be a non-uniform DIF. In the with group membership. DIF is a necessary
middle graph the item i is easer in group r at condition for item bias, but is not a sufficient
all levels of θ, just like in the leftmost graph, condition for item bias. Flagging an item as
however the magnitude of discrepancy in biased requires an expert analysis of the item
difficulty conditional on θ is different at dif- content in the context of possible causes of
ferent levels of θ. The rightmost graph pres- DIF. It is possible, that the item-specific fac-
ents an interesting case of non-uniform DIF, tor causing DIF is actually an important part
namely, for examinees of ability θ < 0 the item of test’s content domain which is not repre-
is easier for group r, however for examinees of sented in other items of the test, thus inclu-
ability θ > 0 the same item is easier for group f. sion of such an item presents no hazard to
Pioneering work regarding DIF analysis validity and will not discriminate against any
dates back to the 1960s in the USA, when the of the groups (see Zieky, 1993).
need to identify test items biased with regard It is also worth mentioning the difference
to minorities became acute. Hence in DIF between DIF and between-group differences in
analysis a classical unsymmetrical division ability. The very essence of the concept of DIF
into two groups is present: the focal minor- is to distinguish the actual differences in the
ity group and the reference majority group. In ability level between groups from differences
this article group membership indexes f and in the way the item behaves because of factors
r will be used in order to comply with this other than the ability measured by the test as
tradition however it is worth noting that DIF a whole. Conditioning over θ, which is present
analysis in educational studies is often per- in the definition of DIF, indicates that the anal-
formed when the grouping variable divides ysis is performed under control of differences
examinees more evenly and without any ob- in distribution of ability between groups.
vious indication of which group is more likely According to what was stated above it can
to be measured unfairly by the test – gender be concluded that DIF detection for dichoto-
is probably the best example. mously scored items will require analysis of
, 94 Kondratek, Grudniewska
item difficulty conditional on group member- Item response
ship G with statistical control over the ability Group Total
1 0
variable θ. Operationally the level of ability is f p1fm p0fm pfm
usually defined within the test as some form r p1rm p0rm prm
of score obtained from the whole test. The first Total p1m p0m pm
and natural solution for the problem stated
above was to employ the Mantel–Haenszel The MH test is constructed in terms of odds
(MH) test. Very popular in the setting of clini- ratios. The odds of a correct response are the
cal trial data analysis, the MH test allows analy- probability of correct response divided by the
sis of statistical significance of differences in the probability of incorrect response. Hence the
distribution of a dichotomous outcome variable ratio of such odds for examinees from groups
between two groups stratified on some addi- r and f within score category m is given as:
tional variable that is significantly related to the
outcome. The MH test is also called Cochran–
–Mantel–Haenszel test, to credit Cochran
who proposed a very similar procedure earlier Following the above designations the null
(Agresti, 2002). An alternative approach to DIF hypothesis and the alternative hypothesis of
analysis, to be covered in the article arose with MH test can be stated as (c.f. Dorans and
the rapid development of item response theory Holland, 1993):
(IRT) in the last decades of the 20th century.
In IRT the relationship between item response
and the ability level is modelled explicitly. The null hypothesis states that the odds of
The article begins with a brief introduc- correct response to the item are the same in
tion to both methods of DIF analysis together both groups in every score category m. It can
with specific DIF effect-size measures that be equivalently rewritten in the manner that
can be constructed in each of the approaches. DIF was defined in eq. (1):
Determining the actual magnitude of DIF is
of no less practical importance than signifi- This means that probability of correct
cance analysis, hence the effect-size topic will response to the item is not related to group
organise the logic of presentation of the meth- membership as long as the test score m is tak-
ods. Afterwards, a Monte Carlo experiment en into account. What is unique to the MH test
comparing the performance of both methods is the way the alternative hypothesis is stated.
under various conditions is presented. The H1 of the MH test states that the difference
of these conditional-on-the-score category
Mantel–Haenszel test DIF analysis probabilities will be nonzero in a constant di-
rection. Moreover, according to H1 all the odds
In the MH approach responses to a dichot- ratios αm will equal one common odds ratio α.
omous item analysed for DIF between two The number of observations can be indexed
groups are stratified on the number of sum analogously as the earlier probabilities:
score points obtained in the test. A contin-
Item response
gency table of size 2 × 2 × M is thus obtained, Group Total
where M is the total number of sum score 0 1
points. Probabilities of observing a particu- r N0rm N1rm Nrm
lar response to the item conditional on group f N0fm N1fm Nfm
membership and sum score category m are
Total N0m N1m Nm
denoted in the following manner:
An interdisciplinary approach
ISSN 0239-6858
Comparison of Mantel–Haenszel with IRT
procedures for DIF detection and effect size
estimation for dichotomous items
Bartosz Kondratek, Magdalena Grudniewska
Student Performance Analysis Unit, Educational Research Institute*
he article compares two methods used to detect diferential item functioning (DIF) of dichotomously scored
items: a nonparametric solution based on the Mantel–Haenszel procedure (MH) and a parametric IRT ap-
proach with a likelihood ratio test. A Monte Carlo experiment was performed in order to evaluate performance
of both statistics in various conditions of DIF uniformity. Results conirmed the theoretical prediction that the
MH test has greater statistical power in detecting uniform DIF than the likelihood ratio test and less power than
the LR test in cases of non-uniform DIF. Apart of examining statistical power of the test, speciic measures of
DIF efect size were compared: MH D–DIF and three measures of P–DIF expressed on the item easiness scale.
Keywords: diferential item functioning, Mantel–Haenszel test, item response theory.
D ifferential item functioning (DIF) is a sta-
tistical term that describes a condition
when the item response is dependent not only
G be the group membership variable, then
the general equation defining DIF with re-
gard to group membership will be of the form
on the ability level of the subject but also on the (c.f. Penfield and Camilli, 2007):
value of some additional group membership
variable. If DIF is present verification plays an which states that conditional distribution of
important role in psychometric analysis of a test the response is not explained solely by the
and is closely related to the problem of validity. ability variable (θ) of the examinee but is
Let Ui denote response to item i, θ be the additionally related to which group (G) the
level of ability that the test measures and examinee belongs to. In the case of dichoto-
mously scored item it can be rewritten as:
Article based on research carried out within the systemic which means that the probability of correct
project “Quality and effectiveness of education – strength-
ening of institutional research capabilities” executed by the response to the item is dependent not only
Educational Research Institute and co-financed from the on the ability θ, but also on the group mem-
European Social Fund (Human Capital Operational Pro- bership G. If G is two valued, G ∈ {f, r}, then
gramme 2007–2013, Priority III High quality of the edu- differential item functioning of item i can be
cation system). A preliminary version of this article was
also expressed as:
published primarily in Polish in Edukacja, 122(2) 2013.
*
Address: Zespół Analiz Osiągnięć Uczniów, Instytut (1)
Badań Edukacyjnych, ul. Górczewska 8, 01-180 Warszawa, meaning that the probability of correct re-
Poland. Email: sponse of an examinee with ability θ in group
, Comparison of Mantel–Haenszel with IRT procedures for DIF detection 93
uniform DIF non-uniform DIF non-uniform DIF
1
1
1
.8
.8
.8
P(U_i=1|theta)
P(U_i=1|theta)
P(U_i=1|theta)
.6
.6
.6
.4
.4
.4
.2
.2
.2
0
0
0
-4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4
theta theta theta
Figure 1. Examples of DIF (coninuous line depicts for: G = r, dashed line for: G = f).
f differs from the probability of correct re- The term item bias refers to the situation
sponse of an examinee with the same level of when one group is being favoured over an-
ability in group r. other group as a consequence of item con-
Figure 1 collects three examples of differ- tent unrelated to the ability intended to be
ential item functioning as defined by eq. (1) measured by the test. Item bias is a specific
by means of curves that depict the probabili- distortion of validity of the test and is not
ty of correct response conditional on θ (item equivalent to the presence of DIF. Differential
characteristic curve, ICC). The left-hand item functioning states that the item response
graph shows the uniform DIF – the ICC in is related to some additional factor that at the
one group is shifted parallel to the ICC for same time is unrelated to the ability mea-
the same item in the other group. All other sured by the test as a whole but is correlated
cases of DIF will be a non-uniform DIF. In the with group membership. DIF is a necessary
middle graph the item i is easer in group r at condition for item bias, but is not a sufficient
all levels of θ, just like in the leftmost graph, condition for item bias. Flagging an item as
however the magnitude of discrepancy in biased requires an expert analysis of the item
difficulty conditional on θ is different at dif- content in the context of possible causes of
ferent levels of θ. The rightmost graph pres- DIF. It is possible, that the item-specific fac-
ents an interesting case of non-uniform DIF, tor causing DIF is actually an important part
namely, for examinees of ability θ < 0 the item of test’s content domain which is not repre-
is easier for group r, however for examinees of sented in other items of the test, thus inclu-
ability θ > 0 the same item is easier for group f. sion of such an item presents no hazard to
Pioneering work regarding DIF analysis validity and will not discriminate against any
dates back to the 1960s in the USA, when the of the groups (see Zieky, 1993).
need to identify test items biased with regard It is also worth mentioning the difference
to minorities became acute. Hence in DIF between DIF and between-group differences in
analysis a classical unsymmetrical division ability. The very essence of the concept of DIF
into two groups is present: the focal minor- is to distinguish the actual differences in the
ity group and the reference majority group. In ability level between groups from differences
this article group membership indexes f and in the way the item behaves because of factors
r will be used in order to comply with this other than the ability measured by the test as
tradition however it is worth noting that DIF a whole. Conditioning over θ, which is present
analysis in educational studies is often per- in the definition of DIF, indicates that the anal-
formed when the grouping variable divides ysis is performed under control of differences
examinees more evenly and without any ob- in distribution of ability between groups.
vious indication of which group is more likely According to what was stated above it can
to be measured unfairly by the test – gender be concluded that DIF detection for dichoto-
is probably the best example. mously scored items will require analysis of
, 94 Kondratek, Grudniewska
item difficulty conditional on group member- Item response
ship G with statistical control over the ability Group Total
1 0
variable θ. Operationally the level of ability is f p1fm p0fm pfm
usually defined within the test as some form r p1rm p0rm prm
of score obtained from the whole test. The first Total p1m p0m pm
and natural solution for the problem stated
above was to employ the Mantel–Haenszel The MH test is constructed in terms of odds
(MH) test. Very popular in the setting of clini- ratios. The odds of a correct response are the
cal trial data analysis, the MH test allows analy- probability of correct response divided by the
sis of statistical significance of differences in the probability of incorrect response. Hence the
distribution of a dichotomous outcome variable ratio of such odds for examinees from groups
between two groups stratified on some addi- r and f within score category m is given as:
tional variable that is significantly related to the
outcome. The MH test is also called Cochran–
–Mantel–Haenszel test, to credit Cochran
who proposed a very similar procedure earlier Following the above designations the null
(Agresti, 2002). An alternative approach to DIF hypothesis and the alternative hypothesis of
analysis, to be covered in the article arose with MH test can be stated as (c.f. Dorans and
the rapid development of item response theory Holland, 1993):
(IRT) in the last decades of the 20th century.
In IRT the relationship between item response
and the ability level is modelled explicitly. The null hypothesis states that the odds of
The article begins with a brief introduc- correct response to the item are the same in
tion to both methods of DIF analysis together both groups in every score category m. It can
with specific DIF effect-size measures that be equivalently rewritten in the manner that
can be constructed in each of the approaches. DIF was defined in eq. (1):
Determining the actual magnitude of DIF is
of no less practical importance than signifi- This means that probability of correct
cance analysis, hence the effect-size topic will response to the item is not related to group
organise the logic of presentation of the meth- membership as long as the test score m is tak-
ods. Afterwards, a Monte Carlo experiment en into account. What is unique to the MH test
comparing the performance of both methods is the way the alternative hypothesis is stated.
under various conditions is presented. The H1 of the MH test states that the difference
of these conditional-on-the-score category
Mantel–Haenszel test DIF analysis probabilities will be nonzero in a constant di-
rection. Moreover, according to H1 all the odds
In the MH approach responses to a dichot- ratios αm will equal one common odds ratio α.
omous item analysed for DIF between two The number of observations can be indexed
groups are stratified on the number of sum analogously as the earlier probabilities:
score points obtained in the test. A contin-
Item response
gency table of size 2 × 2 × M is thus obtained, Group Total
where M is the total number of sum score 0 1
points. Probabilities of observing a particu- r N0rm N1rm Nrm
lar response to the item conditional on group f N0fm N1fm Nfm
membership and sum score category m are
Total N0m N1m Nm
denoted in the following manner:
