International Journal of Innovation in Science and Mathematics Education, 26(1), 1-15, 2018.
Development and Validation of an Item-bank
for an Adaptive Computer-based Test
Assessing K-4 Students’ Mathematical
Competence
Macarena Larraina, Marianela Navarroa, Micaela Buraschia, Isabel Torresa and Kattia Muñoza
Corresponding author: Macarena Larrain ()
a
Universidad de los Andes, Chile
Keywords: elementary mathematics, formative assessment, computer-based test, item validation,
mathematics adaptive assessment
International Journal of Innovation in Science and Mathematics Education, 26(1), 1–15, 2018.
Abstract
This paper reports the development and validation process of an item-bank for an adaptive computer-based test aimed
at assessing kindergarten to fourth grade students’ mathematical competence. A total of 528 multiple-choice items for
44 learning objectives were selected from the Quantile Framework Database, adapted and translated. A sample of 5,145
Chilean students from kindergarten to fourth grade attending 22 schools participated in the statistical validation process.
Item facility and discrimination indices were analysed. Their distributions within grade levels and the four content
domains were explored. Results confirm that a local validation of items was necessary.
Introduction
People deal with diverse mathematical situations daily, all of which they have to learn to interpret
so as to be able to comprehend and analyse the information they perceive (National Council of
Teachers of Mathematics [NCTM] & National Association for the Education of Young Children
[NAEYC], 2002). The world in which we live is a quantitative world, making it necessary to have
adequate comprehension of mathematics to understand what happens around us. This is why
Villarroel (2009) states that cultural adaptability depends mainly on an individual's mathematical
competence, due to the fact that they need to carry out such ordinary tasks as paying accounts, going
shopping, following recipes and sticking to a budget, among others.
The relevance of learning mathematics in early stages
Learning mathematics is a crucial part of development, because it allows students to master skills
that are not acquired through other subjects. The tasks students face while learning mathematics
range from algorithmic procedures to solving word problems, where reasoning and following logical
steps are the means of reaching an answer. Mathematics requires that students learn to identify
relevant information and to choose appropriate strategies to solve problems (Hefty, 2015), which
1
, International Journal of Innovation in Science and Mathematics Education, 26(1), 1-15, 2018.
also contributes to structuring students’ logical thinking. In this sense, the development of
mathematical competence helps people make better judgments as they are based on reasoning
through logical steps using quantitative data, helping make decisions in a constructive, reflexive and
committed manner (Agencia de Calidad de la Educación, 2014).
Additionally, as stated by Clements and Sarama (2009) ‘in a global economy with the vast majority
of jobs requiring more sophisticated skills than in the past, […] educators, communities and business
leaders have expressed strong concern about students’ mathematics achievement’ (p. 4), recognising
its increasing importance for the individual’s subsequent professional achievement and their role in
society (Kilpatrick, Swafford, & Findell, 2001; Doig, McCrae, & Rowe, 2003).
Detecting difficulties and implementing strategies on time
To learn new concepts and skills in mathematics, the student builds on those he has already mastered.
Thus, if a student does not fully understand a previous concept or skill, he will begin to lag behind,
making future progress more difficult (Hunting, Mousley, & Perry, 2012). Therefore, one of the
main reasons for the recent surge of attention to mathematics in early childhood is that early
knowledge strongly affects later success in mathematics (Denton & West, 2002). Thus, it is not an
overstatement to say that what children know early, affects them for many years thereafter (Horne,
2005; National Mathematics Advisory Panel [NMP], 2008).
The early diagnosis of mathematical difficulties has become an essential task, since it allows teachers
to build a profile about the student’s skill development and identify weaknesses that must be
addressed to promote further progress (Taylor-Cox & Oberdorf, 2013). The diagnosis is thus crucial
for designing and providing teaching strategies that may help students overcome their difficulties
and improve their perspectives. The moment in which this process takes place is also decisive. If a
learning difficulty is not addressed promptly, it is more likely to be sustained in time (Horne, 2005).
Furthermore, when teachers plan promptly, they are more likely to make effective pedagogical
decisions to compensate for the diagnosed deficiencies on time. Thus, it is crucial to assess and
analyse assessment results as early as possible to make sure the correct issues are being undertaken.
By periodically receiving information regarding their students’ abilities, teachers are able to adjust
their plans and strategies to meet the children’s needs, taking into consideration what challenges
they will be able to face without frustration (Taylor-Cox & Oberdorf, 2013).
Finally, implementing strategies to address early difficulties in a timely manner will also foster
students’ motivation, as they will feel better prepared to deal with mathematical problems. They will
understand that there is no such thing as a lack of mathematical aptitude (Hunting et al., 2012), and
that mathematical competence is a skill in constant development that needs to be practiced before it
is mastered (Agencia de Calidad de la Educación, 2016). Children need to view mathematics as
sensible, useful, and worthwhile and view themselves as capable of thinking mathematically
(Clements & Sarama, 2009).
In order for this to happen, it is important to count on relatively automated and reliable instruments
that will allow teachers to periodically evaluate their students’ progress without demanding an
excessive amount of their time and effort (Singleton, 2001). Rather than spending their time marking
assessments, early childhood and elementary teachers should focus their expertise on designing
pedagogical strategies to meet the individual needs of their students.
2
Development and Validation of an Item-bank
for an Adaptive Computer-based Test
Assessing K-4 Students’ Mathematical
Competence
Macarena Larraina, Marianela Navarroa, Micaela Buraschia, Isabel Torresa and Kattia Muñoza
Corresponding author: Macarena Larrain ()
a
Universidad de los Andes, Chile
Keywords: elementary mathematics, formative assessment, computer-based test, item validation,
mathematics adaptive assessment
International Journal of Innovation in Science and Mathematics Education, 26(1), 1–15, 2018.
Abstract
This paper reports the development and validation process of an item-bank for an adaptive computer-based test aimed
at assessing kindergarten to fourth grade students’ mathematical competence. A total of 528 multiple-choice items for
44 learning objectives were selected from the Quantile Framework Database, adapted and translated. A sample of 5,145
Chilean students from kindergarten to fourth grade attending 22 schools participated in the statistical validation process.
Item facility and discrimination indices were analysed. Their distributions within grade levels and the four content
domains were explored. Results confirm that a local validation of items was necessary.
Introduction
People deal with diverse mathematical situations daily, all of which they have to learn to interpret
so as to be able to comprehend and analyse the information they perceive (National Council of
Teachers of Mathematics [NCTM] & National Association for the Education of Young Children
[NAEYC], 2002). The world in which we live is a quantitative world, making it necessary to have
adequate comprehension of mathematics to understand what happens around us. This is why
Villarroel (2009) states that cultural adaptability depends mainly on an individual's mathematical
competence, due to the fact that they need to carry out such ordinary tasks as paying accounts, going
shopping, following recipes and sticking to a budget, among others.
The relevance of learning mathematics in early stages
Learning mathematics is a crucial part of development, because it allows students to master skills
that are not acquired through other subjects. The tasks students face while learning mathematics
range from algorithmic procedures to solving word problems, where reasoning and following logical
steps are the means of reaching an answer. Mathematics requires that students learn to identify
relevant information and to choose appropriate strategies to solve problems (Hefty, 2015), which
1
, International Journal of Innovation in Science and Mathematics Education, 26(1), 1-15, 2018.
also contributes to structuring students’ logical thinking. In this sense, the development of
mathematical competence helps people make better judgments as they are based on reasoning
through logical steps using quantitative data, helping make decisions in a constructive, reflexive and
committed manner (Agencia de Calidad de la Educación, 2014).
Additionally, as stated by Clements and Sarama (2009) ‘in a global economy with the vast majority
of jobs requiring more sophisticated skills than in the past, […] educators, communities and business
leaders have expressed strong concern about students’ mathematics achievement’ (p. 4), recognising
its increasing importance for the individual’s subsequent professional achievement and their role in
society (Kilpatrick, Swafford, & Findell, 2001; Doig, McCrae, & Rowe, 2003).
Detecting difficulties and implementing strategies on time
To learn new concepts and skills in mathematics, the student builds on those he has already mastered.
Thus, if a student does not fully understand a previous concept or skill, he will begin to lag behind,
making future progress more difficult (Hunting, Mousley, & Perry, 2012). Therefore, one of the
main reasons for the recent surge of attention to mathematics in early childhood is that early
knowledge strongly affects later success in mathematics (Denton & West, 2002). Thus, it is not an
overstatement to say that what children know early, affects them for many years thereafter (Horne,
2005; National Mathematics Advisory Panel [NMP], 2008).
The early diagnosis of mathematical difficulties has become an essential task, since it allows teachers
to build a profile about the student’s skill development and identify weaknesses that must be
addressed to promote further progress (Taylor-Cox & Oberdorf, 2013). The diagnosis is thus crucial
for designing and providing teaching strategies that may help students overcome their difficulties
and improve their perspectives. The moment in which this process takes place is also decisive. If a
learning difficulty is not addressed promptly, it is more likely to be sustained in time (Horne, 2005).
Furthermore, when teachers plan promptly, they are more likely to make effective pedagogical
decisions to compensate for the diagnosed deficiencies on time. Thus, it is crucial to assess and
analyse assessment results as early as possible to make sure the correct issues are being undertaken.
By periodically receiving information regarding their students’ abilities, teachers are able to adjust
their plans and strategies to meet the children’s needs, taking into consideration what challenges
they will be able to face without frustration (Taylor-Cox & Oberdorf, 2013).
Finally, implementing strategies to address early difficulties in a timely manner will also foster
students’ motivation, as they will feel better prepared to deal with mathematical problems. They will
understand that there is no such thing as a lack of mathematical aptitude (Hunting et al., 2012), and
that mathematical competence is a skill in constant development that needs to be practiced before it
is mastered (Agencia de Calidad de la Educación, 2016). Children need to view mathematics as
sensible, useful, and worthwhile and view themselves as capable of thinking mathematically
(Clements & Sarama, 2009).
In order for this to happen, it is important to count on relatively automated and reliable instruments
that will allow teachers to periodically evaluate their students’ progress without demanding an
excessive amount of their time and effort (Singleton, 2001). Rather than spending their time marking
assessments, early childhood and elementary teachers should focus their expertise on designing
pedagogical strategies to meet the individual needs of their students.
2
