Cohen’s kappa calculator: how far raters agree beyond chance

Two raters who label at random will still agree some of the time. Jacob Cohen’s 1960 coefficient takes that out: it asks how much of the agreement that was possible beyond chance was actually achieved.

A worked example

Two reviewers mark 50 proposals yes or no: both yes 20, yes and no 5, no and yes 10, both no 15. They agree on 35, so pₒ = 0.70. Reviewer A said yes to 25 of 50 and B to 30 of 50, so chance agreement is 0.5 × 0.6 + 0.5 × 0.4 = 0.50. Then κ = (0.70 − 0.50) / (1 − 0.50) = 0.40. Choose Cross-table and paste 20 5 and 10 15 on two lines to see the same number.

The interval uses the large-sample variance of Fleiss, Cohen and Everitt (1969), which is the one to use for a non-zero kappa; the test against chance uses the variance under the null hypothesis, so the two standard errors differ.

The labels almost everyone quotes come from Landis and Koch (1977), who themselves called the divisions arbitrary. McHugh (2012) argues they are too generous for health research, where a wrong rating has consequences, and proposes a stricter reading:

Many published coding studies aim for 0.60 or higher and treat 0.80 as the target for a coding scheme you rely on. Whatever threshold you use, three things move kappa independently of how good the raters are:

• Number of categories. More categories usually mean a lower kappa for the same raters, so scales of different length are not directly comparable.
• Prevalence. When one category dominates, chance agreement is high and kappa falls.
• Weights. A quadratic-weighted kappa is typically higher than the unweighted one on the same data. State which one you report.

Load the example called “The paradox”: two raters pass or fail 50 items, agree on 46 of them, and kappa is only 0.29. At 92 % agreement, that looks like a bug. It is not.

Both raters said “pass” 94 % of the time, so they would agree about 89 % of the time by chance alone. The 92 % is only a little better than that. Feinstein and Cicchetti described the effect in 1990, and it is why a single kappa can mislead on imbalanced data. For that case the page shows three further numbers:

• Maximum kappa is the highest value possible given how often each rater used each category. If it is far below 1, the raters’ different base rates cap the result.
• PABAK (Byrt, Bishop and Carlin, 1993) sets chance to a uniform spread over the categories; here it is 0.84. With two categories it is simply 2pₒ − 1.
• Gwet’s AC1 (2008) models chance differently and stays stable when prevalence is extreme; here it is 0.91.

None of these is a way to rescue a bad number. The point of Feinstein and Cicchetti’s advice is to report agreement and kappa side by side and let a reader see why they differ. Also check that the rare category has enough cases: the agreement on the items that matter may rest on only a handful of them.

Linear or quadratic? Quadratic weights penalise big disagreements much more and, with balanced marginals, are close to an intraclass correlation. Pick the scheme before looking at the results, since it can shift the value by a band.

Fleiss with two raters is not Cohen. Fleiss’ kappa generalises Scott’s pi: it assumes a common distribution of categories across raters, while Cohen’s keeps a separate one per rater. Give this page two rater columns and it calculates Cohen’s. Fleiss’ kappa also does not need the same people to rate every subject, only the same number.

Which rater is it? A single Fleiss value cannot say. The pair table can: in the “Four raters” example, Fleiss’ kappa is 0.38, Dev’s mean kappa with the others is about zero, and without Dev’s column the same data give 0.79. Light’s kappa, the mean of the pairwise values, is a common alternative summary for this kind of data.

Whenever more than one person turns qualitative material into categories, kappa tells you whether the categories exist outside one person’s head. That covers tagging support tickets, coding interview transcripts, rating the severity of usability problems and labelling data for a model.

• Pilot the codebook. Have two people code the same 30 to 50 items before the real pass. In the “Two raters” example, kappa is 0.76 overall but “howto” sits at 0.60, because the raters disagree about what counts as a bug. That is a fix to the definition, not a problem with the raters.
• Severity ratings need weights. Calling a problem “major” when your colleague says “critical” is not the same as calling it “cosmetic”. Unweighted kappa treats them alike.
• Standard scales are a different question. For a questionnaire’s internal consistency, use the Cronbach’s alpha calculator; for usability scores, the SUS calculator.

1. Cohen J. “A coefficient of agreement for nominal scales.” Educational and Psychological Measurement 20(1), 1960, 37–46.
2. Cohen J. “Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit.” Psychological Bulletin 70(4), 1968, 213–220.
3. Fleiss J. L., Cohen J., Everitt B. S. “Large sample standard errors of kappa and weighted kappa.” Psychological Bulletin 72(5), 1969, 323–327.
4. Cicchetti D. V., Allison T. “A new procedure for assessing reliability of scoring EEG sleep recordings.” American Journal of EEG Technology 11(3), 1971, 101–109 — linear weights.
5. Fleiss J. L. “Measuring nominal scale agreement among many raters.” Psychological Bulletin 76(5), 1971, 378–382.
6. Fleiss J. L., Nee J. C. M., Landis J. R. “Large sample variance of kappa in the case of different sets of raters.” Psychological Bulletin 86(5), 1979, 974–977.
7. Landis J. R., Koch G. G. “The measurement of observer agreement for categorical data.” Biometrics 33(1), 1977, 159–174.
8. Feinstein A. R., Cicchetti D. V. “High agreement but low kappa: I. The problems of two paradoxes.” Journal of Clinical Epidemiology 43(6), 1990, 543–549.
9. Byrt T., Bishop J., Carlin J. B. “Bias, prevalence and kappa.” Journal of Clinical Epidemiology 46(5), 1993, 423–429.
10. Gwet K. L. “Computing inter-rater reliability and its variance in the presence of high agreement.” British Journal of Mathematical and Statistical Psychology 61(1), 2008, 29–48.
11. McHugh M. L. “Interrater reliability: the kappa statistic.” Biochemia Medica 22(3), 2012, 276–282.