Clew

HomeToolsUX researchIntraclass correlation calculator

Intraclass correlation calculator: can you trust a score from these raters?

Work out the intraclass correlation coefficient without SPSS or an Excel ANOVA. Paste one row per subject and one column per rater, answer three questions about your study, and get the ICC you should report, with its confidence interval, the other five forms next to it, and the rater who is pulling the number down.

● Free, no sign-upUpdated:

ICC

Paste from Excel, Google Sheets or a CSV export. A header row and an id column are recognised. Rows with an empty score are left out as a whole.
Who rated the subjects?
What counts as agreeing?
What will you use in practice?
Confidence level

—

 

ICC— 
Interval— 
F test— 
SEM— 
Raters for 0.75— 
Subjects × raters— 
ICC and its interval on the Koo & Li scale
ICC and its interval on the Koo & Li scale
ICC and its confidence interval

All six forms

Each intraclass correlation form with its meaning, value, confidence interval, F test and p value; the chosen form is highlighted
FormWhat it answersICCIntervalF (df)p

Each rater

Mean and standard deviation of each rater, distance from the overall mean, and the chosen ICC with that rater left out
RaterMeanSDvs overallICC without

ANOVA table

Sums of squares, degrees of freedom and mean squares, with the share of variance from each two-way component
SourceSSdfMSVariance

Method. Mean squares from a one-way and a two-way ANOVA without replication, then the formulas of Shrout & Fleiss (1979) and McGraw & Wong (1996). Intervals are F-based; the absolute-agreement forms use Satterthwaite’s degrees of freedom. SEM = SD × √(1 − ICC), using the single-rater ICC and the SD of all scores. Rows with a missing score are dropped.

Everything is calculated in your browser — nothing you enter is sent anywhere.

How to use it

  1. Paste one column per raterCopy the block from your spreadsheet with the header row. Cells must be numbers. An id column at the start is recognised and left out.
  2. Say who ratedDifferent raters for each subject is the one-way model. If the same raters scored everything, say whether they stand in for raters in general or whether only these people matter.
  3. Say what agreeing meansIf 70 and 60 should count as disagreement, choose same score. If only the order matters, because a fixed offset can be calibrated out, choose same ranking.
  4. Say what you will usePick one rater’s score if one person will rate from now on, and the mean only if the real score really will be the average of this many raters.
  5. Report the form and the intervalWrite the model, type and definition with the value and its 95 % interval, as Koo and Li ask. “ICC = 0.87” alone does not tell a reader which of six numbers that is.

How the intraclass correlation is calculated

An ICC is a share of variance: how much of the spread in the scores comes from real differences between subjects, rather than from the raters and from noise. It starts from an ANOVA on the table of scores.

n subjects, k raters MSR between subjects MSC between raters MSE residual MSW within subjects ICC(1,1) = (MSR − MSW) / (MSR + (k − 1)·MSW) ICC(2,1) = (MSR − MSE) / (MSR + (k − 1)·MSE + k·(MSC − MSE)/n) ICC(3,1) = (MSR − MSE) / (MSR + (k − 1)·MSE) ICC(1,k) = (MSR − MSW) / MSR ICC(2,k) = (MSR − MSE) / (MSR + (MSC − MSE)/n) ICC(3,k) = (MSR − MSE) / MSR

A worked example

Load “Shrout & Fleiss”: the six subjects and four judges from their 1979 paper. The mean squares are MSR = 11.24, MSW = 6.26, MSC = 32.49 and MSE = 1.02. The same table gives ICC(1,1) = 0.17, ICC(2,1) = 0.29 and ICC(3,1) = 0.71, and for the mean of four judges 0.44, 0.62 and 0.91. The judges rank the subjects quite alike but sit at very different levels, so the choice of form changes the answer from poor to good. That is why you have to choose the form before you look at the numbers.

Which ICC should I use?

Koo and Li’s guideline reduces the choice to the three questions in the form above.

Which ICC form fits the study
StudyModelReport
Each subject scored by different, randomly assigned ratersOne-way randomICC(1,1), or ICC(1,k) if you use the mean
Same raters for everyone, and results should carry over to similar ratersTwo-way randomICC(2,1), usually absolute agreement
Same raters for everyone, and only these raters matter; or test-retestTwo-way mixedICC(3,1) for consistency, ICC(A,1) for absolute agreement

Two-way random and two-way mixed give the same number for the same definition. The formulas do not differ; what you may claim does. Under the random model the result generalises to other raters like these. Under the mixed model it describes only these raters.

Absolute agreement or consistency? Load “Three raters”: Dev scores every recording about 12 points below Ana and Ben, but in the same order. Consistency, ICC(C,1), is 0.98. Absolute agreement, ICC(A,1), is 0.87, with an interval from 0.21 to 0.96. Both are correct: they answer different questions.

What is a good intraclass correlation coefficient?

Two scales are in common use. Koo and Li (2016) is the stricter and the more recent; Cicchetti (1994) is the one many older papers quote.

Two common readings of an ICC
ICCKoo & Li (2016)Cicchetti (1994)
below 0.40PoorPoor
0.40 – 0.49PoorFair
0.50 – 0.59ModerateFair
0.60 – 0.74ModerateGood
0.75 – 0.89GoodExcellent
0.90 and aboveExcellentExcellent

Koo and Li apply their labels to the 95 % interval, not to the point estimate: an interval from 0.83 to 0.94 is “good to excellent”. The three-rater example above has a point estimate that reads “good” and an interval that also reaches “poor”.

A low ICC does not always mean poor raters. Since the ICC compares differences between subjects with everything else, the same raters score lower on subjects who are all alike. If you test only on experienced users, or only on similar pages, you can get a low number from a perfectly careful team. Sample subjects who cover the range you will meet in practice.

The interval, SEM and how many raters you need

The interval usually matters more than the ICC. With few subjects it is very wide, and with few raters the absolute-agreement form is wider still, since rater variance is estimated from only a few raters. In our own simulation (30 subjects, raters contributing a sizeable share of the variance), the nominal 95 % interval for ICC(A,1) covered the true value about 90 % of the time with 3 raters, 93 % with 5 and 94 % with 10. The consistency and one-way intervals held 95 %. So read the limits of the absolute form as approximate when raters are few.

Standard error of measurement turns the ICC into the units of your scale. SEM = SD × √(1 − ICC), and the smallest change you can tell from noise is about 1.96 × √2 × SEM (Weir, 2005). In the “Test-retest” example, 15 users take the SUS twice: ICC is 0.95 and SEM 2.9, so a user’s score has to move by about 8 points before it is distinguishable from noise. SEM depends on the SD of your sample, so it does not carry over directly to a very different group.

Raters needed. The Spearman-Brown formula forecasts what averaging would do: m = target × (1 − ICC) / (ICC × (1 − target)), rounded up, starting from the single-rater ICC. It is a forecast from the point estimate, so it assumes that new raters behave like these.

Calculating ICC in Excel, SPSS and R

Excel has no ICC function. Enable the Analysis ToolPak, run “Anova: Two-Factor Without Replication” on the table, and put the mean squares into the formulas above. That gives the two-way forms; the one-way form needs its own ANOVA, and the intervals need F quantiles with non-integer degrees of freedom. Pasting the table here gives all of that at once.

SPSS: Analyze → Scale → Reliability Analysis → Statistics → Intraclass correlation coefficient. Choose the model and type. “Single Measures” is the row for one rater, and “Average Measures” the row for the mean.

R: psych::ICC() prints all six forms, and irr::icc() gives one at a time.

The same forms under different names
Shrout & FleissMcGraw & WongSPSS
ICC(1,1) · ICC(1,k)ICC(1) · ICC(k)One-Way Random
ICC(2,1) · ICC(2,k)ICC(A,1) · ICC(A,k)Two-Way, Absolute Agreement
ICC(3,1) · ICC(3,k)ICC(C,1) · ICC(C,k)Two-Way, Consistency

ICC, kappa, Cronbach’s alpha and Pearson

  • Categories rather than numbers? Use the Cohen’s kappa calculator. Quadratic-weighted kappa on an ordered scale behaves much like an ICC.
  • Questionnaire items rather than raters? ICC(3,k) is numerically the same as Cronbach’s alpha, so the Cronbach’s alpha calculator gives that number along with the item analysis you will want for a scale.
  • Why not Pearson’s r? It compares two columns at a time and ignores level. A rater who scores everyone exactly 20 points lower has r = 1. ICC(A,1) sees the offset.

For product and UX work, the ICC answers the question behind any numeric judgement call: when several people score session recordings, rate the quality of support replies or grade design reviews, is the score a property of the thing being rated or of the person rating it?

Sources

  1. Shrout P. E., Fleiss J. L. “Intraclass correlations: uses in assessing rater reliability.” Psychological Bulletin 86(2), 1979, 420–428.
  2. McGraw K. O., Wong S. P. “Forming inferences about some intraclass correlation coefficients.” Psychological Methods 1(1), 1996, 30–46.
  3. Koo T. K., Li M. Y. “A guideline of selecting and reporting intraclass correlation coefficients for reliability research.” Journal of Chiropractic Medicine 15(2), 2016, 155–163.
  4. Cicchetti D. V. “Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.” Psychological Assessment 6(4), 1994, 284–290.
  5. Weir J. P. “Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM.” Journal of Strength and Conditioning Research 19(1), 2005, 231–240.

FAQ

What is the intraclass correlation coefficient?

A measure of how much of the variation in numeric scores comes from real differences between the subjects, rather than from the raters and noise. Close to 1, the score belongs to the subject; close to 0, it mostly reflects who rated and chance.

What is a good ICC value?

By Koo and Li (2016), below 0.50 is poor, 0.50–0.75 moderate, 0.75–0.90 good and above 0.90 excellent, read from the 95 % confidence interval. Cicchetti (1994) is more generous and calls 0.75 and above excellent.

Which ICC should I report?

It depends on who rated, what counts as agreement, and whether you will use one rater’s score or a mean. Most inter-rater studies with the same raters for every subject report ICC(2,1) with absolute agreement. Choose before looking at the results, and name the form when you report it.

What is the difference between ICC(2,1) and ICC(3,1)?

ICC(2,1) counts differences in level between raters as error, so it measures absolute agreement. ICC(3,1) removes them, so it measures consistency. The gap between them shows how much raters differ in level.

How do I calculate ICC in Excel?

Run “Anova: Two-Factor Without Replication” from the Analysis ToolPak and put the mean squares into the formulas on this page. The interval takes further F quantiles. Or paste the same table here to get all six forms with intervals.

How many subjects and raters do I need?

Koo and Li recommend at least 30 varied subjects and at least 3 raters. Fewer still produce a number, but with a wide interval, which this page shows.

Why is my ICC negative?

Because subjects differ less than raters disagree. It tends to happen with small samples or very similar subjects. It is an estimate, not a real negative reliability, and it should be read as no reliability.

Is my data uploaded anywhere?

No. Parsing and every calculation run in your browser. Only what you put into the URL yourself, by copying the share link, leaves the page.

More in UX research

Open the collection →

Tours, tooltips and checklists whose impact shows up in the numbers.

Try Clew for freeHow it worksFree plan forever · no credit card

Product tours, popups and onboarding in the age of AI

7 patterns with step counts, copy rules and what to measure, what AI changes, and a checklist before you publish. PDF, 2 pages. What is inside →