Available online at www.sciencedirect.com
Procedia - Social and Behavioral Sciences 59 (2012) 297 – 303
UKM Teaching and Learning Congress 2011
Automated analysis of exam questions according to bloom’s
taxonomy
Nazlia Omara, , Syahidah Sufi Harisa, Rosilah Hassana, Haslina Arshada, Masura
Rahmata, Noor Faridatul Ainun Zainala & Rozli Zulkiflib
a
Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia
b
Faculty of Engineering and Build Environment, Universiti Kebangsaan Malaysia
Abstract
Bloom's Taxonomy is a classification of learning objectives within education that educators set for students. The cognitive
domain within this taxonomy is designed to verify a student's cognitive level during a written examination. Educators may
sometimes face the challenge in analysing whether their examination questions comply within the requirements of the Bloom’s
taxonomy at different cognitive levels. This paper proposes an automated analysis of the exam questions to determine the
appropriate category based on this taxonomy. This rule-based approach applies Natural Language Processing (NLP) techniques
to identify important keywords and verbs, which may assist in the identification of the category of a question. This work focuses
on the computer programming subject domain. At present, a set of 100 questions (70 training set and 30 test set) is used in the
research. Preliminary results indicate that the rules may successfully assist in the identification of the Bloom’s taxonomy
category correctly in the exam questions.
© 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning
© 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and
CongressCongress
Learning 2011. 2011
Keywords: Bloom’s taxonomy; natural language processing; rule-based
1. Introduction
There are many types of assessment or 'testing' to access student's learning curves. However, written examination
is the most common approach used by any higher education institutions for students' assessment. Question is an
element that is intertwined with the examination. Questions raised in the paper plays an important role in efforts to
test the students' overall cognitive levels held each semester. Effective style of questioning as described by Swart
(2010) is always an issue to help students attend to the desired learning outcome. Furthermore, to make it effective,
balancing between lower and higher-level question is a must Swart (2010). Bloom's Taxonomy, created by Bloom
(1956), has been widely accepted as a guideline in designing reasonable examination questions belonging to various
cognitive levels. The hierarchical models of Bloom’s are widely used in education fields (Chang & Chung, 2009)
constructing questions (Lister & Leaney, 2003), to ensure balancing and student cognitive mastery (Oliver et al.,
* Corresponding author. Tel.: +6-03- 8921-6733; fax: +6-03-8925-6184
E-mail address:
1877-0428 © 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning Congress 2011
doi:10.1016/j.sbspro.2012.09.278
, 298 Nazlia Omar et al. / Procedia - Social and Behavioral Sciences 59 (2012) 297 – 303
2004). From the computer science domain itself, the taxonomy improves curricular design and assessments (Scoot,
2003).
Normally, academicians would categorise a question according to the Bloom’s cognitive level manually.
However, according to Yusof and Chai (2010), not all can identify the cognitive level of a question correctly. This
may lead to miscategorizing of the exam questions and subsequently may fail to meet the examination standard
required for the subject. In addition, some academicians also show no significant agreement on how to use Bloom's
taxonomy in educating students (Johnson & Fuller, 2006).
The aim of this paper is to propose a rule-based approach in determining the Bloom’s taxonomy cognitive level
of examination questions through natural language processing. Exam questions will be analyzed and each question
will be categorized based on the Bloom’s taxonomy cognitive level. The scope of the work is limited to computer
programming domain. This will assist the academicians in setting up suitable exam questions according to the
requirements.
2. Related Work
Much work (Swart, 2010; Scott, 2003; Thompson et al., 2008; Chang & Chung, 2009) has attempted to classify
exam questions based on the Bloom’s taxonomy. However, there has not been much attempt in using natural
language processing techniques to solve this problem. Chang & Chung (2009) presented an online test system to
classify and analyse the cognitive level of Bloom’s taxonomy to English questions. The system accepts the exam
question as an input, which will then be segmented. This system has a database where various verbs of Bloom's
taxonomy are stored. The database includes verbs with lower-case and capital letters. The system then compares all
the verb tenses present in the questions. When a keyword is found in the test item, then the particular question
belongs to the keyword. Weightage for the question is applied if any of Bloom's category shares the same verb. The
authors provide four match situations to indicate matching items; Correct Match Items, Partial Match Items, No
Keyword items and No Match Items. Result shows that keywords show efficiency only to 'Knowledge' level of
Bloom's.
Previous researchers proposed a model to classify question items with artificial neural network approach that
applies different feature method (Yusof & Chai, 2010). The model is trained using the scaled conjugate gradient
learning algorithm. Several data processing techniques are applied to a feature set and then the content of a question
is transformed into a numeric form called a feature vector. In order to perform text classification, three types of
feature set are used i.e. whole feature set, the Document Frequency (DF) and Category Frequency-Document
Frequency (CF-DF). A question item which consist of 274 questions were selected for processing. From the system,
out of the three feature sets, DF reductions gave more efficient result with the combination of classification and
convergence time.
Automarking (Cutrone & Chang, 2010), a learning management system, is capable of automarking once students
submit their answers online. Through natural language processing, the student's answer is evaluated with semantic
meaning. This is done through text pre-processing phase where the semantic meaning get 'special space'. The
product of pre-processing phase is the canonical form. Comparisons between the canonical from the student’s
response and the correct answer are compared to achieve the level of equivalence. Finally, appropriate grade values
will be given. However, the system is unable to analyse multiple sentences based on the overall meaning.
Although all the works above incorporate Bloom's taxonomy in their work, they do not categorise question based
on the semantic of the text. A work from Chang and Chung (2009) is based on keyword matching while keywords are
varied over researchers. Question categorization should imply the nature of the question and how the questions can
help educators to identify the learner's cognitive level.
3. Bloom’s Cognitive Domain
Cognitive domain of Bloom's Taxonomy (Bloom, 1956) is one of the three domains that were introduced by
Benjamin Bloom in 1950s. This domain is designed to verify a student's cognitive quality during written
examination. The famous Bloom's taxonomy consists of six levels i.e. knowledge, comprehension, application,
analysis, synthesis and evaluation (Bloom, 1956).
The following describe each levels of Bloom's Taxonomy:
Procedia - Social and Behavioral Sciences 59 (2012) 297 – 303
UKM Teaching and Learning Congress 2011
Automated analysis of exam questions according to bloom’s
taxonomy
Nazlia Omara, , Syahidah Sufi Harisa, Rosilah Hassana, Haslina Arshada, Masura
Rahmata, Noor Faridatul Ainun Zainala & Rozli Zulkiflib
a
Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia
b
Faculty of Engineering and Build Environment, Universiti Kebangsaan Malaysia
Abstract
Bloom's Taxonomy is a classification of learning objectives within education that educators set for students. The cognitive
domain within this taxonomy is designed to verify a student's cognitive level during a written examination. Educators may
sometimes face the challenge in analysing whether their examination questions comply within the requirements of the Bloom’s
taxonomy at different cognitive levels. This paper proposes an automated analysis of the exam questions to determine the
appropriate category based on this taxonomy. This rule-based approach applies Natural Language Processing (NLP) techniques
to identify important keywords and verbs, which may assist in the identification of the category of a question. This work focuses
on the computer programming subject domain. At present, a set of 100 questions (70 training set and 30 test set) is used in the
research. Preliminary results indicate that the rules may successfully assist in the identification of the Bloom’s taxonomy
category correctly in the exam questions.
© 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning
© 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and
CongressCongress
Learning 2011. 2011
Keywords: Bloom’s taxonomy; natural language processing; rule-based
1. Introduction
There are many types of assessment or 'testing' to access student's learning curves. However, written examination
is the most common approach used by any higher education institutions for students' assessment. Question is an
element that is intertwined with the examination. Questions raised in the paper plays an important role in efforts to
test the students' overall cognitive levels held each semester. Effective style of questioning as described by Swart
(2010) is always an issue to help students attend to the desired learning outcome. Furthermore, to make it effective,
balancing between lower and higher-level question is a must Swart (2010). Bloom's Taxonomy, created by Bloom
(1956), has been widely accepted as a guideline in designing reasonable examination questions belonging to various
cognitive levels. The hierarchical models of Bloom’s are widely used in education fields (Chang & Chung, 2009)
constructing questions (Lister & Leaney, 2003), to ensure balancing and student cognitive mastery (Oliver et al.,
* Corresponding author. Tel.: +6-03- 8921-6733; fax: +6-03-8925-6184
E-mail address:
1877-0428 © 2011 Published by Elsevier Ltd. Selection and/or peer reviewed under responsibility of the UKM Teaching and Learning Congress 2011
doi:10.1016/j.sbspro.2012.09.278
, 298 Nazlia Omar et al. / Procedia - Social and Behavioral Sciences 59 (2012) 297 – 303
2004). From the computer science domain itself, the taxonomy improves curricular design and assessments (Scoot,
2003).
Normally, academicians would categorise a question according to the Bloom’s cognitive level manually.
However, according to Yusof and Chai (2010), not all can identify the cognitive level of a question correctly. This
may lead to miscategorizing of the exam questions and subsequently may fail to meet the examination standard
required for the subject. In addition, some academicians also show no significant agreement on how to use Bloom's
taxonomy in educating students (Johnson & Fuller, 2006).
The aim of this paper is to propose a rule-based approach in determining the Bloom’s taxonomy cognitive level
of examination questions through natural language processing. Exam questions will be analyzed and each question
will be categorized based on the Bloom’s taxonomy cognitive level. The scope of the work is limited to computer
programming domain. This will assist the academicians in setting up suitable exam questions according to the
requirements.
2. Related Work
Much work (Swart, 2010; Scott, 2003; Thompson et al., 2008; Chang & Chung, 2009) has attempted to classify
exam questions based on the Bloom’s taxonomy. However, there has not been much attempt in using natural
language processing techniques to solve this problem. Chang & Chung (2009) presented an online test system to
classify and analyse the cognitive level of Bloom’s taxonomy to English questions. The system accepts the exam
question as an input, which will then be segmented. This system has a database where various verbs of Bloom's
taxonomy are stored. The database includes verbs with lower-case and capital letters. The system then compares all
the verb tenses present in the questions. When a keyword is found in the test item, then the particular question
belongs to the keyword. Weightage for the question is applied if any of Bloom's category shares the same verb. The
authors provide four match situations to indicate matching items; Correct Match Items, Partial Match Items, No
Keyword items and No Match Items. Result shows that keywords show efficiency only to 'Knowledge' level of
Bloom's.
Previous researchers proposed a model to classify question items with artificial neural network approach that
applies different feature method (Yusof & Chai, 2010). The model is trained using the scaled conjugate gradient
learning algorithm. Several data processing techniques are applied to a feature set and then the content of a question
is transformed into a numeric form called a feature vector. In order to perform text classification, three types of
feature set are used i.e. whole feature set, the Document Frequency (DF) and Category Frequency-Document
Frequency (CF-DF). A question item which consist of 274 questions were selected for processing. From the system,
out of the three feature sets, DF reductions gave more efficient result with the combination of classification and
convergence time.
Automarking (Cutrone & Chang, 2010), a learning management system, is capable of automarking once students
submit their answers online. Through natural language processing, the student's answer is evaluated with semantic
meaning. This is done through text pre-processing phase where the semantic meaning get 'special space'. The
product of pre-processing phase is the canonical form. Comparisons between the canonical from the student’s
response and the correct answer are compared to achieve the level of equivalence. Finally, appropriate grade values
will be given. However, the system is unable to analyse multiple sentences based on the overall meaning.
Although all the works above incorporate Bloom's taxonomy in their work, they do not categorise question based
on the semantic of the text. A work from Chang and Chung (2009) is based on keyword matching while keywords are
varied over researchers. Question categorization should imply the nature of the question and how the questions can
help educators to identify the learner's cognitive level.
3. Bloom’s Cognitive Domain
Cognitive domain of Bloom's Taxonomy (Bloom, 1956) is one of the three domains that were introduced by
Benjamin Bloom in 1950s. This domain is designed to verify a student's cognitive quality during written
examination. The famous Bloom's taxonomy consists of six levels i.e. knowledge, comprehension, application,
analysis, synthesis and evaluation (Bloom, 1956).
The following describe each levels of Bloom's Taxonomy:
