ICC(1,C)

- Defined by Shrout and Fleiss (1979) and McGraw and Wong (1996)

- Calculate the degree of consistency among single measurements rated by randomly selected raters.

Interpretation:

The degree of consistency among single measurements rated by randomly selected raters. Note that this has the same formula as ICC(3,1,C), however the assumption is different as this is assuming raters are randomly selected as opposed to a specific set of raters.
Koo and Li (2016) gives the following suggestion for interpreting ICC (Koo and Li 2016):
below 0.50: poor
between 0.50 and 0.75: moderate
between 0.75 and 0.90: good
above 0.90: excellent

Underlying Model: Two-way random effects model

\(x_{ij} = \mu + r_i + c_j + e_{ij} \)

Assumptions:

\(\mu\): The population mean of observations

\(r_i\): The row effects (effects between each sample) are random, independent and normally distributed with mean 0 and variance \(\sigma_{r}^2\).

\(c_i\): The row effects (effects between each rater) are random, independent and normally distributed with mean 0 and variance \(\sigma_{c}^2\).

\(e_{ij}\): The residual error are random, independent and normally distributed with mean 0 and variance \(\sigma_{e}^2\). All residual effects are pairwise independent.

Formula:

\(\frac{MS_R - MS_E}{MS_R + (k-1) MS_E}\)

\(MS_R\) = mean square for rows;

\(MS_C\) = mean square for columns;

\(MS_E\) = mean square error;

k = number of measurements (number of columns);

Example in R

Untitled.utf8 
library("irr")
## Loading required package: lpSolve
data("anxiety", package = "irr")
anxiety
##    rater1 rater2 rater3
## 1       3      3      2
## 2       3      6      1
## 3       3      4      4
## 4       4      6      4
## 5       5      2      3
## 6       5      4      2
## 7       2      2      1
## 8       3      4      6
## 9       5      3      1
## 10      2      3      1
## 11      2      2      1
## 12      6      3      2
## 13      1      3      3
## 14      5      3      3
## 15      2      2      1
## 16      2      2      1
## 17      1      1      3
## 18      2      3      3
## 19      4      3      2
## 20      3      4      2

Use irr Library, need to specify model, type, unit:

icc(
    anxiety, model = "twoway", 
    type = "consistency", unit = "single"
)
##  Single Score Intraclass Correlation
## 
##    Model: twoway 
##    Type : consistency 
## 
##    Subjects = 20 
##      Raters = 3 
##    ICC(C,1) = 0.216
## 
##  F-Test, H0: r0 = 0 ; H1: r0 > 0 
##    F(19,38) = 1.83 , p = 0.0562 
## 
##  95%-Confidence Interval for ICC Population Values:
##   -0.046 < ICC < 0.522

Use psych Library, calculate all type at once, for ICC(1,C), read from “Single_fixed_raters”:

library(psych)
ICC(anxiety)
## Call: ICC(x = anxiety)
## 
## Intraclass correlation coefficients 
##                          type  ICC   F df1 df2     p lower bound upper bound
## Single_raters_absolute   ICC1 0.18 1.6  19  40 0.094     -0.0405        0.44
## Single_random_raters     ICC2 0.20 1.8  19  38 0.056     -0.0045        0.45
## Single_fixed_raters      ICC3 0.22 1.8  19  38 0.056     -0.0073        0.48
## Average_raters_absolute ICC1k 0.39 1.6  19  40 0.094     -0.1323        0.70
## Average_random_raters   ICC2k 0.43 1.8  19  38 0.056     -0.0136        0.71
## Average_fixed_raters    ICC3k 0.45 1.8  19  38 0.056     -0.0222        0.73
## 
##  Number of subjects = 20     Number of Judges =  3

Live Example: Try this yourself!

References:

Koo, Terry, and Mae Li. 2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (March). doi:10.1016/j.jcm.2016.02.012.
Shrout, P.E., and J.L. Fleiss. 1979. “Intraclass Correlation: Uses in Assessing Rater Reliability.” Psychological Bulletin 86: 420–28. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1:30–46. [Google Scholar]