Evaluation of Academic Activities in Universities
VALIDITY, RELIABILITY AND DIFFICULTY INDICES FOR
INSTRUCTOR-BUILT EXAM QUESTIONS1
Gholamreza JANDAGHI2
PhD, Associate Professor Faculty of Management, University of Tehran, Qom Campus, Iran
E-mail:
Fatemeh SHATERIAN3
MSc, Academic member of Islamic Azad University, Saveh Branch, Iran
E-mail:
Abstract: The purpose of the research is to determine college Instructor’s skill rate in
designing exam questions in chemistry subject. The statistical population was all of chemistry
exam shits for two semesters in one academic year from which a sample of 364 exam shits
was drawn using multistage cluster sampling. Two experts assessed the shits and by using
appropriate indices and z-test and chi-squared test the analysis of the data was done. We
found that the designed exams have suitable coefficients of validity and reliability. The level of
difficulty of exams was high. No significant relationship was found between male and female
instructors in terms of the coefficient of validity and reliability but a significant difference
between the difficulty level in male and female instructors was found(P<.001). It means that
female instructors had designed more difficult questions. We did not find any significant
relationship between the instructors’ gender and the coefficient of discrimination of the exams.
Key words: instructor-built exam; content validity; face validity; reliability; coefficient of
discrimination; coefficient of difficulty
1. Introduction
Examination and testing is an important part of a teaching-learning process which
allows instructors to evaluate their students during and at the end of an educational course.
Many instructors dislike preparing and grading exams, and most students dread taking them.
Yet tests are powerful educational tools that serve at least four functions. First, tests help you
151
, Evaluation of Academic Activities in Universities
evaluate students and assess whether they are learning what you are expecting them to
learn. Second, well-designed tests serve to motivate and help students structure their
academic efforts. Crooks (1988), McKeachie (1986), and Wergin (1988) report that students
study in ways that reflect how they think they will be tested. In last 40 years the most exams
used to evaluate the students have been designed by instructors. Some may have used tests
which have been designed by outsider exam designers. These tests have not had enough
efficiency (Seif 2004). Regarding the importance of instructor-designed test in evaluation
process of the students, many researches have been done in this area (Lotfabadi 1997). In
theory, the best test for a subject is a test that includes all educational objectives of the
course. But if the test is too long, its preparation is impractical. Therefore, instead of
including all content and objectives, one may choose some questions which are
representative of the whole subject to achieve all objectives. Such a test is said to have
content validity (Seif 2004).
Content validity of a instructor-designed test can be assessed by a sample of the
test questions. When a test does not have content validity two possible outcomes may occur.
First, the students can not present the skills that are not included in the test when they need.
Second, instead some unrelated question may be included in the test that are answered
wrongly. The important point here is that we should not mistake the face validity with
content validity. Basically the face validity is a measure that determines whether a test is
measuring logically and whether students think the test questions are appropriate ( Lotfabadi
1997).
Based on what is said, an ideal test in addition to measuring what is supposed to
measure, must be consistently constant in different times. This characteristic is called
reliability. Other measures of an ideal test are difficulty level and discriminant index. The
total percent of the individuals who answer the question correctly is known as difficulty
coefficient denoted by P (Seif 2004). The discriminant index is a measure of discrimination
between strong and weak groups. In this study, we intend to evaluate the extent of ideal
quality measures (validity, reliability,…) in instructor-designed test for first year college.
Materials and methods
The statistical population in this study consisted of all chemistry exam papers for
final chemistry exams in first and second semester for first year of college in Qom province
of Iran of which a sample of 364 was taken. A twostage cluster sampling was used to draw
samples. In first stage three colleges was randomly selected. In second stage a number of
exam papers from each college was selected according to the number of students in each
college.
In this study the content validity of the exam questions was assessed in two ways. In
the first method we used a two dimensional table. One dimension was educational goals
and the second dimension was the content of the course materials(Seif 2004). The second
method applied for assessing content validity was a questionnaire with Likert scale in which
two chemistry education expert evaluated the extent of compatibility of exam questions with
course contents. For assessment of face validity of instructor-built exams we used a 12-item
questionnaire answered by two chemistry experts.
152
VALIDITY, RELIABILITY AND DIFFICULTY INDICES FOR
INSTRUCTOR-BUILT EXAM QUESTIONS1
Gholamreza JANDAGHI2
PhD, Associate Professor Faculty of Management, University of Tehran, Qom Campus, Iran
E-mail:
Fatemeh SHATERIAN3
MSc, Academic member of Islamic Azad University, Saveh Branch, Iran
E-mail:
Abstract: The purpose of the research is to determine college Instructor’s skill rate in
designing exam questions in chemistry subject. The statistical population was all of chemistry
exam shits for two semesters in one academic year from which a sample of 364 exam shits
was drawn using multistage cluster sampling. Two experts assessed the shits and by using
appropriate indices and z-test and chi-squared test the analysis of the data was done. We
found that the designed exams have suitable coefficients of validity and reliability. The level of
difficulty of exams was high. No significant relationship was found between male and female
instructors in terms of the coefficient of validity and reliability but a significant difference
between the difficulty level in male and female instructors was found(P<.001). It means that
female instructors had designed more difficult questions. We did not find any significant
relationship between the instructors’ gender and the coefficient of discrimination of the exams.
Key words: instructor-built exam; content validity; face validity; reliability; coefficient of
discrimination; coefficient of difficulty
1. Introduction
Examination and testing is an important part of a teaching-learning process which
allows instructors to evaluate their students during and at the end of an educational course.
Many instructors dislike preparing and grading exams, and most students dread taking them.
Yet tests are powerful educational tools that serve at least four functions. First, tests help you
151
, Evaluation of Academic Activities in Universities
evaluate students and assess whether they are learning what you are expecting them to
learn. Second, well-designed tests serve to motivate and help students structure their
academic efforts. Crooks (1988), McKeachie (1986), and Wergin (1988) report that students
study in ways that reflect how they think they will be tested. In last 40 years the most exams
used to evaluate the students have been designed by instructors. Some may have used tests
which have been designed by outsider exam designers. These tests have not had enough
efficiency (Seif 2004). Regarding the importance of instructor-designed test in evaluation
process of the students, many researches have been done in this area (Lotfabadi 1997). In
theory, the best test for a subject is a test that includes all educational objectives of the
course. But if the test is too long, its preparation is impractical. Therefore, instead of
including all content and objectives, one may choose some questions which are
representative of the whole subject to achieve all objectives. Such a test is said to have
content validity (Seif 2004).
Content validity of a instructor-designed test can be assessed by a sample of the
test questions. When a test does not have content validity two possible outcomes may occur.
First, the students can not present the skills that are not included in the test when they need.
Second, instead some unrelated question may be included in the test that are answered
wrongly. The important point here is that we should not mistake the face validity with
content validity. Basically the face validity is a measure that determines whether a test is
measuring logically and whether students think the test questions are appropriate ( Lotfabadi
1997).
Based on what is said, an ideal test in addition to measuring what is supposed to
measure, must be consistently constant in different times. This characteristic is called
reliability. Other measures of an ideal test are difficulty level and discriminant index. The
total percent of the individuals who answer the question correctly is known as difficulty
coefficient denoted by P (Seif 2004). The discriminant index is a measure of discrimination
between strong and weak groups. In this study, we intend to evaluate the extent of ideal
quality measures (validity, reliability,…) in instructor-designed test for first year college.
Materials and methods
The statistical population in this study consisted of all chemistry exam papers for
final chemistry exams in first and second semester for first year of college in Qom province
of Iran of which a sample of 364 was taken. A twostage cluster sampling was used to draw
samples. In first stage three colleges was randomly selected. In second stage a number of
exam papers from each college was selected according to the number of students in each
college.
In this study the content validity of the exam questions was assessed in two ways. In
the first method we used a two dimensional table. One dimension was educational goals
and the second dimension was the content of the course materials(Seif 2004). The second
method applied for assessing content validity was a questionnaire with Likert scale in which
two chemistry education expert evaluated the extent of compatibility of exam questions with
course contents. For assessment of face validity of instructor-built exams we used a 12-item
questionnaire answered by two chemistry experts.
152
