Psyc 516 Chapter 6 - Reliability
Item Analysis ✔️Examining whether the test/measure did what it should have done
* Discriminate between those who know the material and those who do not.
* Often used with multiple choice tests
* Includes item difficulty, test difficulty, and item discrimination
Quantitatively examining individual items within a test
* See which items aren't functioning right, poorly worded. Re-write or drop items
Item Difficulty ✔️Percentage (or proportion) of people who got the item correct
* The easier the item, the larger this will be
* If no one passes or everyone passes - item is no good because there is no information about individual
differences
* Want items with a moderate spread of difficulty (avg of .50)
Test Difficulty ✔️Examines difficulty of the test as a whole
* Plot distribution of total scores
* Depends on the difficulty of items that make up the test
* Typically modified until normal curve is approximated - by adding/deleting items or use of scoring
weights (e.g. curving a test)
Test difficulty: Piling at the low end ✔️Too high a floor - too hard of a test
Lacks sufficient number of easy items to properly discriminate at the low end
Test difficulty: Piling at the high end ✔️Too low of a ceiling - too easy of a test
Lacks sufficient number of difficult items to properly discriminate at the high end.
,How difficult should a test be? ✔️Depends on the purpose of testing.
* Screening purposes = difficulty values should be close to selection ratio. Want the top 20% - want
difficulty level of .20 (or higher for guessing)
* Standardized tests are generally designed to elicit maximum differentiation among individuals at all
levels
* Most tests are medium difficulty = .50
If the purpose is to admit people into grad school, have a higher threshold. In this class, high
understanding of course content.
If purpose is to screen for who gets into grad school, and we only want 10%, then we want high
difficulty.
Maximize individual differences. We want to look at across everyone, differentiate among who learned
the content better.
Item Discrimination ✔️Degree to which an item differentiates correctly among test takers in the
behavior/content that the test is designed to measure
* Multiple-choice: how effectively item discriminates between high and low groups
* Common practice: compare upper group and lower groups selected from the extremes of the
distribution
* Optimum percentage is between 25% and 33 % in each group
Accurately differentiate among test takers in content that the test is designed to measure. When items
are taken out b/c low scorers got it wrong but high scorers got it right as opposed to "80% got it wrong."
Index of Discrimination ✔️When there is a right answer - like exam scores, not likert scales
Calculate the percentage of people who responded correctly in the upper group = 4/16 got an A (25%).
3/4 upper group got question right (75%)
then same for lower group 4/16 got a C (25%). Only 1/4 lower group got question right (25%)
, *D=U-L
75-25=50 - on higher end
* Ranges from -100 to +100
** + 100 means ALL of the U group and none of the L group got the answer right
Positive values indicating that people in the high group got the answer right low group got the answer
wrong, negative means more people in the lower group got the item correct than the higher group
Relation between Item difficulty and Item discrimination ✔️Perfect test has two characteristics -
* Sufficiently difficulty such that 50% of all test takers get it correct
* All test takers in the U group get item correct and all test takers in the L group get the item wrong
What constrains item discrimination? ✔️Item difficulty.
Perfect discrimination when difficulty = .50
The more difficult, the less the item discriminates
Item Discrimination in Scale Development: Initial Stages ✔️Total scores are the closest approximation
of the construct under examination
* Maximize internal consistency ---- Include a lot of items, and see how well it correlates with other
items as well as the total score
Remove or revise items that have low correlations with total scores
*Retains items with the highest average intercorrelations
*Purifies or homogenizes the scale/test
We don't look at items individually; look at total scores for the best approximation of construct
Item Analysis ✔️Examining whether the test/measure did what it should have done
* Discriminate between those who know the material and those who do not.
* Often used with multiple choice tests
* Includes item difficulty, test difficulty, and item discrimination
Quantitatively examining individual items within a test
* See which items aren't functioning right, poorly worded. Re-write or drop items
Item Difficulty ✔️Percentage (or proportion) of people who got the item correct
* The easier the item, the larger this will be
* If no one passes or everyone passes - item is no good because there is no information about individual
differences
* Want items with a moderate spread of difficulty (avg of .50)
Test Difficulty ✔️Examines difficulty of the test as a whole
* Plot distribution of total scores
* Depends on the difficulty of items that make up the test
* Typically modified until normal curve is approximated - by adding/deleting items or use of scoring
weights (e.g. curving a test)
Test difficulty: Piling at the low end ✔️Too high a floor - too hard of a test
Lacks sufficient number of easy items to properly discriminate at the low end
Test difficulty: Piling at the high end ✔️Too low of a ceiling - too easy of a test
Lacks sufficient number of difficult items to properly discriminate at the high end.
,How difficult should a test be? ✔️Depends on the purpose of testing.
* Screening purposes = difficulty values should be close to selection ratio. Want the top 20% - want
difficulty level of .20 (or higher for guessing)
* Standardized tests are generally designed to elicit maximum differentiation among individuals at all
levels
* Most tests are medium difficulty = .50
If the purpose is to admit people into grad school, have a higher threshold. In this class, high
understanding of course content.
If purpose is to screen for who gets into grad school, and we only want 10%, then we want high
difficulty.
Maximize individual differences. We want to look at across everyone, differentiate among who learned
the content better.
Item Discrimination ✔️Degree to which an item differentiates correctly among test takers in the
behavior/content that the test is designed to measure
* Multiple-choice: how effectively item discriminates between high and low groups
* Common practice: compare upper group and lower groups selected from the extremes of the
distribution
* Optimum percentage is between 25% and 33 % in each group
Accurately differentiate among test takers in content that the test is designed to measure. When items
are taken out b/c low scorers got it wrong but high scorers got it right as opposed to "80% got it wrong."
Index of Discrimination ✔️When there is a right answer - like exam scores, not likert scales
Calculate the percentage of people who responded correctly in the upper group = 4/16 got an A (25%).
3/4 upper group got question right (75%)
then same for lower group 4/16 got a C (25%). Only 1/4 lower group got question right (25%)
, *D=U-L
75-25=50 - on higher end
* Ranges from -100 to +100
** + 100 means ALL of the U group and none of the L group got the answer right
Positive values indicating that people in the high group got the answer right low group got the answer
wrong, negative means more people in the lower group got the item correct than the higher group
Relation between Item difficulty and Item discrimination ✔️Perfect test has two characteristics -
* Sufficiently difficulty such that 50% of all test takers get it correct
* All test takers in the U group get item correct and all test takers in the L group get the item wrong
What constrains item discrimination? ✔️Item difficulty.
Perfect discrimination when difficulty = .50
The more difficult, the less the item discriminates
Item Discrimination in Scale Development: Initial Stages ✔️Total scores are the closest approximation
of the construct under examination
* Maximize internal consistency ---- Include a lot of items, and see how well it correlates with other
items as well as the total score
Remove or revise items that have low correlations with total scores
*Retains items with the highest average intercorrelations
*Purifies or homogenizes the scale/test
We don't look at items individually; look at total scores for the best approximation of construct
