Task 4 – Measurement error and sources of variation
1. What is reliability?
Reliability is defined as the degree to which measurement is free from measurement error.
general definition
In addition to the general definition, there is an extended definition, which is the extent to
which scores for patients who have not changed are the same for repeated measurements
under several conditions:
- Internal consistency = using different sets of items from the same multi-item
measurement
- Test-retest = over time
- Inter-rater = by different person on the same occasion
- Intra-rater = by the same persons on different occasions
TEST-RETEST RELIABILITY
Test-retest reliability evaluates the consistency of the measuring instrument, which is the
probability of producing the same results with repeated administration of the test. This type
of reliability is useful in situations where clinicians are involved in the measurement process,
such as self-administered questionnaires. Evaluating test-retest reliability involves a sample
of patients completing the test on two occasions, under the same conditions. If the test is
reliable, each patient’s score will be identical across the two occasions. The period of time
between occasions of testing should be carefully considered. It should be far enough apart
to avoid fatigue, learning or memory effects but close enough to avoid changes in the
measures variable.
INTRA-RATER RELIABILITY
Intra-rater reliability refers to the consistency of measurement recorded by one rater across
two (or more) occasions under the same conditions using the same standardized protocols
and equipment. If the test is 100% reliable, each patient’s score will be identical on both
occasions of testing, assuming no clinical change in their status. In reality, errors occur even
when measurements are undertaken by one clinician. The period of time between
measurements is important to considers.
INTER-RATER RELIABILITY
Inter-rater reliability refers to the consistency of measurement recorded by two (or more)
raters on the same occasion. This is the probability of different clinicians producing the same
results on the same patients. It involves two clinicians taking measurements on one cohort
of patients under the same conditions using the same standardized protocols and
equipment. If the test is reliable, both clinician’s scores will be identical for each individual
patient.
Misunderstanding: subjective measurements are less reliable than objective
measurements
Measurement instruments often contain multiple items to assess subjective
constructs increases their reliability.
, Task 4 – Measurement error and sources of variation
The true score is the average score that would be obtained if the measurements were
performed an infinite number of times.
Observed score =
Y = observed score
N = true score
E = error term of the measurement
In the classical test theory the assumptions is that the error terms are uncorrelated with the
true score and uncorrelated with each other.
Variance of the observed scores =
Sigma Y = total variance
Sigma N = true variance
Sigma e = error variance
An additional assumption is that error variances ae constant for every repetitions. This
implies that the total variance is also constant.
Reliability as the proportion of the total variance in the measurements, which is due to true
differences between the patients (true variance), the reliability parameter can be
represented by:
Rel = observed variance / total variance
The reliability parameter expresses how well patients can be distinguished form each other
despite the presence of measurement error.
From this formula, one can also calculate the standard error of measurement (SEM) as a
parameter of measurement error, which is:
Standard deviation (SD) = between patient variation
Reliability parameters range in value from 0 (totally unreliable) to 1 (perfect reliable). If the
measurement error is small in comparison with variability between patients, the reliability
parameter approaches 1.
If the reliability is perfect, we would expect that all the dots are on the 45 line
1. What is reliability?
Reliability is defined as the degree to which measurement is free from measurement error.
general definition
In addition to the general definition, there is an extended definition, which is the extent to
which scores for patients who have not changed are the same for repeated measurements
under several conditions:
- Internal consistency = using different sets of items from the same multi-item
measurement
- Test-retest = over time
- Inter-rater = by different person on the same occasion
- Intra-rater = by the same persons on different occasions
TEST-RETEST RELIABILITY
Test-retest reliability evaluates the consistency of the measuring instrument, which is the
probability of producing the same results with repeated administration of the test. This type
of reliability is useful in situations where clinicians are involved in the measurement process,
such as self-administered questionnaires. Evaluating test-retest reliability involves a sample
of patients completing the test on two occasions, under the same conditions. If the test is
reliable, each patient’s score will be identical across the two occasions. The period of time
between occasions of testing should be carefully considered. It should be far enough apart
to avoid fatigue, learning or memory effects but close enough to avoid changes in the
measures variable.
INTRA-RATER RELIABILITY
Intra-rater reliability refers to the consistency of measurement recorded by one rater across
two (or more) occasions under the same conditions using the same standardized protocols
and equipment. If the test is 100% reliable, each patient’s score will be identical on both
occasions of testing, assuming no clinical change in their status. In reality, errors occur even
when measurements are undertaken by one clinician. The period of time between
measurements is important to considers.
INTER-RATER RELIABILITY
Inter-rater reliability refers to the consistency of measurement recorded by two (or more)
raters on the same occasion. This is the probability of different clinicians producing the same
results on the same patients. It involves two clinicians taking measurements on one cohort
of patients under the same conditions using the same standardized protocols and
equipment. If the test is reliable, both clinician’s scores will be identical for each individual
patient.
Misunderstanding: subjective measurements are less reliable than objective
measurements
Measurement instruments often contain multiple items to assess subjective
constructs increases their reliability.
, Task 4 – Measurement error and sources of variation
The true score is the average score that would be obtained if the measurements were
performed an infinite number of times.
Observed score =
Y = observed score
N = true score
E = error term of the measurement
In the classical test theory the assumptions is that the error terms are uncorrelated with the
true score and uncorrelated with each other.
Variance of the observed scores =
Sigma Y = total variance
Sigma N = true variance
Sigma e = error variance
An additional assumption is that error variances ae constant for every repetitions. This
implies that the total variance is also constant.
Reliability as the proportion of the total variance in the measurements, which is due to true
differences between the patients (true variance), the reliability parameter can be
represented by:
Rel = observed variance / total variance
The reliability parameter expresses how well patients can be distinguished form each other
despite the presence of measurement error.
From this formula, one can also calculate the standard error of measurement (SEM) as a
parameter of measurement error, which is:
Standard deviation (SD) = between patient variation
Reliability parameters range in value from 0 (totally unreliable) to 1 (perfect reliable). If the
measurement error is small in comparison with variability between patients, the reliability
parameter approaches 1.
If the reliability is perfect, we would expect that all the dots are on the 45 line
