HomeToolsUX researchIntraclass correlation calculator
Intraclass correlation calculator: can you trust a score from these raters?
Work out the intraclass correlation coefficient without SPSS or an Excel ANOVA. Paste one row per subject and one column per rater, answer three questions about your study, and get the ICC you should report, with its confidence interval, the other five forms next to it, and the rater who is pulling the number down.
—
All six forms
| Form | What it answers | ICC | Interval | F (df) | p |
|---|
Each rater
| Rater | Mean | SD | vs overall | ICC without |
|---|
ANOVA table
| Source | SS | df | MS | Variance |
|---|
Method. Mean squares from a one-way and a two-way ANOVA without replication, then the formulas of Shrout & Fleiss (1979) and McGraw & Wong (1996). Intervals are F-based; the absolute-agreement forms use Satterthwaite’s degrees of freedom. SEM = SD × √(1 − ICC), using the single-rater ICC and the SD of all scores. Rows with a missing score are dropped.
Everything is calculated in your browser — nothing you enter is sent anywhere.
How to use it
- Paste one column per raterCopy the block from your spreadsheet with the header row. Cells must be numbers. An id column at the start is recognised and left out.
- Say who ratedDifferent raters for each subject is the one-way model. If the same raters scored everything, say whether they stand in for raters in general or whether only these people matter.
- Say what agreeing meansIf 70 and 60 should count as disagreement, choose same score. If only the order matters, because a fixed offset can be calibrated out, choose same ranking.
- Say what you will usePick one rater’s score if one person will rate from now on, and the mean only if the real score really will be the average of this many raters.
- Report the form and the intervalWrite the model, type and definition with the value and its 95 % interval, as Koo and Li ask. “ICC = 0.87” alone does not tell a reader which of six numbers that is.
How the intraclass correlation is calculated
An ICC is a share of variance: how much of the spread in the scores comes from real differences between subjects, rather than from the raters and from noise. It starts from an ANOVA on the table of scores.
A worked example
Load “Shrout & Fleiss”: the six subjects and four judges from their 1979 paper. The mean squares are MSR = 11.24, MSW = 6.26, MSC = 32.49 and MSE = 1.02. The same table gives ICC(1,1) = 0.17, ICC(2,1) = 0.29 and ICC(3,1) = 0.71, and for the mean of four judges 0.44, 0.62 and 0.91. The judges rank the subjects quite alike but sit at very different levels, so the choice of form changes the answer from poor to good. That is why you have to choose the form before you look at the numbers.
Which ICC should I use?
Koo and Li’s guideline reduces the choice to the three questions in the form above.
| Study | Model | Report |
|---|---|---|
| Each subject scored by different, randomly assigned raters | One-way random | ICC(1,1), or ICC(1,k) if you use the mean |
| Same raters for everyone, and results should carry over to similar raters | Two-way random | ICC(2,1), usually absolute agreement |
| Same raters for everyone, and only these raters matter; or test-retest | Two-way mixed | ICC(3,1) for consistency, ICC(A,1) for absolute agreement |
Two-way random and two-way mixed give the same number for the same definition. The formulas do not differ; what you may claim does. Under the random model the result generalises to other raters like these. Under the mixed model it describes only these raters.
Absolute agreement or consistency? Load “Three raters”: Dev scores every recording about 12 points below Ana and Ben, but in the same order. Consistency, ICC(C,1), is 0.98. Absolute agreement, ICC(A,1), is 0.87, with an interval from 0.21 to 0.96. Both are correct: they answer different questions.
What is a good intraclass correlation coefficient?
Two scales are in common use. Koo and Li (2016) is the stricter and the more recent; Cicchetti (1994) is the one many older papers quote.
| ICC | Koo & Li (2016) | Cicchetti (1994) |
|---|---|---|
| below 0.40 | Poor | Poor |
| 0.40 – 0.49 | Poor | Fair |
| 0.50 – 0.59 | Moderate | Fair |
| 0.60 – 0.74 | Moderate | Good |
| 0.75 – 0.89 | Good | Excellent |
| 0.90 and above | Excellent | Excellent |
Koo and Li apply their labels to the 95 % interval, not to the point estimate: an interval from 0.83 to 0.94 is “good to excellent”. The three-rater example above has a point estimate that reads “good” and an interval that also reaches “poor”.
A low ICC does not always mean poor raters. Since the ICC compares differences between subjects with everything else, the same raters score lower on subjects who are all alike. If you test only on experienced users, or only on similar pages, you can get a low number from a perfectly careful team. Sample subjects who cover the range you will meet in practice.
The interval, SEM and how many raters you need
The interval usually matters more than the ICC. With few subjects it is very wide, and with few raters the absolute-agreement form is wider still, since rater variance is estimated from only a few raters. In our own simulation (30 subjects, raters contributing a sizeable share of the variance), the nominal 95 % interval for ICC(A,1) covered the true value about 90 % of the time with 3 raters, 93 % with 5 and 94 % with 10. The consistency and one-way intervals held 95 %. So read the limits of the absolute form as approximate when raters are few.
Standard error of measurement turns the ICC into the units of your scale. SEM = SD × √(1 − ICC), and the smallest change you can tell from noise is about 1.96 × √2 × SEM (Weir, 2005). In the “Test-retest” example, 15 users take the SUS twice: ICC is 0.95 and SEM 2.9, so a user’s score has to move by about 8 points before it is distinguishable from noise. SEM depends on the SD of your sample, so it does not carry over directly to a very different group.
Raters needed. The Spearman-Brown formula forecasts what averaging would do: m = target × (1 − ICC) / (ICC × (1 − target)), rounded up, starting from the single-rater ICC. It is a forecast from the point estimate, so it assumes that new raters behave like these.
Calculating ICC in Excel, SPSS and R
Excel has no ICC function. Enable the Analysis ToolPak, run “Anova: Two-Factor Without Replication” on the table, and put the mean squares into the formulas above. That gives the two-way forms; the one-way form needs its own ANOVA, and the intervals need F quantiles with non-integer degrees of freedom. Pasting the table here gives all of that at once.
SPSS: Analyze → Scale → Reliability Analysis → Statistics → Intraclass correlation coefficient. Choose the model and type. “Single Measures” is the row for one rater, and “Average Measures” the row for the mean.
R: psych::ICC() prints all six forms, and irr::icc() gives one at a time.
| Shrout & Fleiss | McGraw & Wong | SPSS |
|---|---|---|
| ICC(1,1) · ICC(1,k) | ICC(1) · ICC(k) | One-Way Random |
| ICC(2,1) · ICC(2,k) | ICC(A,1) · ICC(A,k) | Two-Way, Absolute Agreement |
| ICC(3,1) · ICC(3,k) | ICC(C,1) · ICC(C,k) | Two-Way, Consistency |
ICC, kappa, Cronbach’s alpha and Pearson
- Categories rather than numbers? Use the Cohen’s kappa calculator. Quadratic-weighted kappa on an ordered scale behaves much like an ICC.
- Questionnaire items rather than raters? ICC(3,k) is numerically the same as Cronbach’s alpha, so the Cronbach’s alpha calculator gives that number along with the item analysis you will want for a scale.
- Why not Pearson’s r? It compares two columns at a time and ignores level. A rater who scores everyone exactly 20 points lower has r = 1. ICC(A,1) sees the offset.
For product and UX work, the ICC answers the question behind any numeric judgement call: when several people score session recordings, rate the quality of support replies or grade design reviews, is the score a property of the thing being rated or of the person rating it?
Sources
- Shrout P. E., Fleiss J. L. “Intraclass correlations: uses in assessing rater reliability.” Psychological Bulletin 86(2), 1979, 420–428.
- McGraw K. O., Wong S. P. “Forming inferences about some intraclass correlation coefficients.” Psychological Methods 1(1), 1996, 30–46.
- Koo T. K., Li M. Y. “A guideline of selecting and reporting intraclass correlation coefficients for reliability research.” Journal of Chiropractic Medicine 15(2), 2016, 155–163.
- Cicchetti D. V. “Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.” Psychological Assessment 6(4), 1994, 284–290.
- Weir J. P. “Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM.” Journal of Strength and Conditioning Research 19(1), 2005, 231–240.
FAQ
What is the intraclass correlation coefficient?
A measure of how much of the variation in numeric scores comes from real differences between the subjects, rather than from the raters and noise. Close to 1, the score belongs to the subject; close to 0, it mostly reflects who rated and chance.
What is a good ICC value?
By Koo and Li (2016), below 0.50 is poor, 0.50–0.75 moderate, 0.75–0.90 good and above 0.90 excellent, read from the 95 % confidence interval. Cicchetti (1994) is more generous and calls 0.75 and above excellent.
Which ICC should I report?
It depends on who rated, what counts as agreement, and whether you will use one rater’s score or a mean. Most inter-rater studies with the same raters for every subject report ICC(2,1) with absolute agreement. Choose before looking at the results, and name the form when you report it.
What is the difference between ICC(2,1) and ICC(3,1)?
ICC(2,1) counts differences in level between raters as error, so it measures absolute agreement. ICC(3,1) removes them, so it measures consistency. The gap between them shows how much raters differ in level.
How do I calculate ICC in Excel?
Run “Anova: Two-Factor Without Replication” from the Analysis ToolPak and put the mean squares into the formulas on this page. The interval takes further F quantiles. Or paste the same table here to get all six forms with intervals.
How many subjects and raters do I need?
Koo and Li recommend at least 30 varied subjects and at least 3 raters. Fewer still produce a number, but with a wide interval, which this page shows.
Why is my ICC negative?
Because subjects differ less than raters disagree. It tends to happen with small samples or very similar subjects. It is an estimate, not a real negative reliability, and it should be read as no reliability.
Is my data uploaded anywhere?
No. Parsing and every calculation run in your browser. Only what you put into the URL yourself, by copying the share link, leaves the page.
More in UX research
- Survey sample size calculator
How many responses does a survey need for a given margin of error?
- Kano model survey analyzer
Which features are must-haves, performance features or delighters for users?
- Van Westendorp calculator
What price range do buyers accept, and where do the price sensitivity meter curves cross?
- Gabor–Granger analysis
At which price on the ladder does revenue peak, and how elastic is demand?
- NPS, CSAT and CES calculator
What is the Net Promoter Score of these answers, with a confidence interval?
- CSAT & CES calculator
What are the CSAT and Customer Effort Score of these survey results?
- SUS calculator
What is the System Usability Scale score, and is the usability good?
- UEQ calculator
How does the product score on the six UEQ scales, and where does it sit against the benchmark?
- NASA-TLX calculator
How heavy did this task feel, and which kind of demand made it heavy?
- Cronbach’s alpha calculator
Do the items of this questionnaire scale hold together, and which item pulls the alpha down?
- Cohen’s kappa calculator
Do two or more raters agree beyond chance, and which label or rater is behind the disagreement?
- Krippendorff’s alpha calculator
Can you rely on this coding when not every coder rated every unit, and which label or coder is behind a low alpha?