CI Exam 3 – Reliability questions with
complete solutions
What is reliability? - correct answer ✔✔ Observed Score = True Score +- Error
Any measurement (score) has 2 components: a true score (T) and error component (e)
*Reliability coefficients tell us how much of the score is the true score and how much is the
error.*
What vales can the reliability coefficient take on Roxx? - correct answer ✔✔ 0 - 1
T/F: Reliability Coefficient tells us correlation and agreement. - correct answer ✔✔ TRUE
Tells us correlation AND agreement.
What does a score of 1 mean?
Give an example with intrareader reliability - correct answer ✔✔ Perfect reliability.
For example with intrareader reliability would take elbow flexion one time, then take it again
and exactly the same, this would be perfect reliability and we would assume there is no source
of error.
What are the three types of reliability? - correct answer ✔✔ 1) Test-Retest
2) Intratester
,3) Intertester
What is the preferred statistic for ratio data? - correct answer ✔✔ Intraclass correlation
coefficient (ICC)
What is the preferred statistic for interval data? - correct answer ✔✔ Intraclass correlation
coefficient (ICC)
What is the preferred statistic for ordinal data? - correct answer ✔✔ Weighted Kappa statistic
The Intra class coefficient (ICC) ranges in value from ___ to ___. - correct answer ✔✔ 0 - 1.0
What things are used to calculate ICC? - correct answer ✔✔ Its a ratio of variances (standard
deviation squared s^2)
using variance estimates obtained through an ANOVA and reflects both correlation and
agreement.
s^2T, s^2F, s^2E
When can ICC be used for ordinal data? - correct answer ✔✔ When intervals between values
are assumed to be equivalent.
(but this is just interval data? apparently there is a test you can do to assume equal intervals for
ordinal data)
How do we obtain our values to calculate ICC? - correct answer ✔✔ Actual calculations require
the use of variance estimates obtained by the ANOVA.
, *We will not be responsible for hand calculating ICC but should understand the different models
and forms and how to interpret them.
s^2T = - correct answer ✔✔ true variance
s^2F = - correct answer ✔✔ variance of facets of interest (characteristics of / are we reliable
between raters, testing conditions, subjects)
What is the preferred statistic for nominal data? - correct answer ✔✔ Cohen's Kappa statistic or
Percent Agreement
s^2F = - correct answer ✔✔ Error variance (unexplained)
How is variance from an ICC calculated different from Variance due to just differences between
observed scores or error? - correct answer ✔✔ It is based on the theory that variance or
differences between observed scores are due to a variety of factors, not just random error.
Differences occur due to characteristics of raters or subjects, testing conditions, or alternate
forms of a test. etc. These factors are now separated from random error and considered factors
in an ANOVA.
Mathmatically, what is the ICC? - correct answer ✔✔ ICC = s^2T / s^2T + s^2F + s^2E
The equation here represents the conceptual basis of the ICC. It is the proportion of the True
Variance divided by the true variance plus Variance attributed to the facets and the error
variance.
If we were testing for inter rater reliability, what is the s^2F? - correct answer ✔✔ s^2F or facet
of interest is rater and the denominator will account for the variance between raters.
complete solutions
What is reliability? - correct answer ✔✔ Observed Score = True Score +- Error
Any measurement (score) has 2 components: a true score (T) and error component (e)
*Reliability coefficients tell us how much of the score is the true score and how much is the
error.*
What vales can the reliability coefficient take on Roxx? - correct answer ✔✔ 0 - 1
T/F: Reliability Coefficient tells us correlation and agreement. - correct answer ✔✔ TRUE
Tells us correlation AND agreement.
What does a score of 1 mean?
Give an example with intrareader reliability - correct answer ✔✔ Perfect reliability.
For example with intrareader reliability would take elbow flexion one time, then take it again
and exactly the same, this would be perfect reliability and we would assume there is no source
of error.
What are the three types of reliability? - correct answer ✔✔ 1) Test-Retest
2) Intratester
,3) Intertester
What is the preferred statistic for ratio data? - correct answer ✔✔ Intraclass correlation
coefficient (ICC)
What is the preferred statistic for interval data? - correct answer ✔✔ Intraclass correlation
coefficient (ICC)
What is the preferred statistic for ordinal data? - correct answer ✔✔ Weighted Kappa statistic
The Intra class coefficient (ICC) ranges in value from ___ to ___. - correct answer ✔✔ 0 - 1.0
What things are used to calculate ICC? - correct answer ✔✔ Its a ratio of variances (standard
deviation squared s^2)
using variance estimates obtained through an ANOVA and reflects both correlation and
agreement.
s^2T, s^2F, s^2E
When can ICC be used for ordinal data? - correct answer ✔✔ When intervals between values
are assumed to be equivalent.
(but this is just interval data? apparently there is a test you can do to assume equal intervals for
ordinal data)
How do we obtain our values to calculate ICC? - correct answer ✔✔ Actual calculations require
the use of variance estimates obtained by the ANOVA.
, *We will not be responsible for hand calculating ICC but should understand the different models
and forms and how to interpret them.
s^2T = - correct answer ✔✔ true variance
s^2F = - correct answer ✔✔ variance of facets of interest (characteristics of / are we reliable
between raters, testing conditions, subjects)
What is the preferred statistic for nominal data? - correct answer ✔✔ Cohen's Kappa statistic or
Percent Agreement
s^2F = - correct answer ✔✔ Error variance (unexplained)
How is variance from an ICC calculated different from Variance due to just differences between
observed scores or error? - correct answer ✔✔ It is based on the theory that variance or
differences between observed scores are due to a variety of factors, not just random error.
Differences occur due to characteristics of raters or subjects, testing conditions, or alternate
forms of a test. etc. These factors are now separated from random error and considered factors
in an ANOVA.
Mathmatically, what is the ICC? - correct answer ✔✔ ICC = s^2T / s^2T + s^2F + s^2E
The equation here represents the conceptual basis of the ICC. It is the proportion of the True
Variance divided by the true variance plus Variance attributed to the facets and the error
variance.
If we were testing for inter rater reliability, what is the s^2F? - correct answer ✔✔ s^2F or facet
of interest is rater and the denominator will account for the variance between raters.
