HomeToolsUX researchKrippendorff’s alpha calculator
Krippendorff’s alpha calculator: can you rely on this coding, gaps included?
Check intercoder reliability without the SPSS macro or a file in a set format. Paste one row per unit and one column per coder from any spreadsheet, with labels or numbers, and leave blank whatever a coder did not rate. You get alpha for the right level of measurement, a bootstrap interval, and the coder and category behind a low number.
—
Every level of measurement
| Level | Fits | Alpha |
|---|
Each coder
| Coder | Rated | Coverage | Alpha without |
|---|
Each category
| Category | Values | Matched | Alpha vs rest |
|---|
Coincidence matrix
Method. Coincidences within units, then α = 1 − Do/De, with the difference function of the chosen level (Krippendorff, 2011). A value alone in its unit cannot be paired and drops out. The interval is a percentile bootstrap: units are resampled whole, up to 1,000 times, with a fixed seed, so a table always gives the same limits. It is not the resampling of the KALPHA macro, so its limits can differ a little.
Everything is calculated in your browser — nothing you enter is sent anywhere.
How to use it
- Paste one column per coderCopy the block from your spreadsheet with the header row. Leave a cell empty, or write NA, where a coder did not rate that unit. An id column at the start is recognised and left out.
- Check the layoutRows are units by default. If each row is one coder, the layout of Krippendorff’s paper and of some R and Python examples, switch it. A verdict that makes no sense is often just this.
- Choose the level of measurementNominal for unordered labels, ordinal for ranks, interval for scores with equal steps, and ratio for counts and durations. Decide from the scale, not from which number looks best.
- Read the interval, not only the pointCompare the lower limit with the threshold you committed to. With few units, a point above 0.80 can come with an interval reaching well below 0.667.
- Find what is behind a low numberLook at the category table first. One weak category usually means the codebook needs a sharper definition; one coder who raises alpha when removed usually means retraining.
How Krippendorff’s alpha is calculated
Alpha compares the disagreement you observed with the disagreement you would expect if values were assigned by chance. It is 1 for perfect agreement and 0 when coders are no better than chance.
The key step is the coincidence matrix. Each unit with m ratings contributes m(m − 1) ordered pairs, and each pair counts 1/(m − 1). So every value counts once, however many coders saw that unit, and a unit with a gap simply contributes fewer pairs.
A worked example
Load “Krippendorff’s example”: 12 units and 4 observers from his own paper, with 7 of the 48 cells empty. Unit 12 has a single value, which cannot be paired, so n = 40. The same table gives 0.743 as nominal data, 0.815 as ordinal, 0.849 as interval and 0.797 as ratio, and this page reproduces all four. Note the interval: with only 11 usable units it runs from 0.40 to 1.00, so those differences between levels are well within sampling noise.
What is a good Krippendorff’s alpha?
Krippendorff’s own guidance is stricter than most kappa scales, and it has two lines rather than a ladder of adjectives.
| Alpha | Reading |
|---|---|
| 0.800 and above | Rely on the variable |
| 0.667 to 0.799 | Only for tentative conclusions |
| below 0.667 | Do not rely on it |
He presents these as conventions and asks that the bar be set by what a wrong conclusion would cost, and fixed before the data arrive. A theme count in an internal research note can live with less than a coding scheme behind a published claim.
Apply the threshold to each variable. Alpha is calculated for one variable at a time, so a codebook with ten fields needs ten alphas, and averaging them hides the weak one. And look at the lower limit. Load “Coded comments”: alpha is 0.72 with an interval from 0.54 to 0.87, so this sample does not rule out either “not reliable” or “reliable”.
Missing data, and why alpha is built for it
Kappa needs every coder on every unit. In practice nobody codes everything: researchers split the load and double-code a sample, or one person leaves halfway through. Most tools then drop any row with a gap.
In “Coded comments” three people code 30 comments, and 14 comments are missing one coder. Dropping those rows would leave 16 comments and throw away 28 ratings that were actually made. Alpha uses all 76 ratings. Zapf and colleagues (2016) found that with ratings missing completely at random alpha stays stable, while running Fleiss’ kappa on the complete rows gives biased estimates.
The catch. This only works if the gaps have nothing to do with the content. If coders skip the units that are hard to code, the hard units never get a chance to disagree, and alpha comes out too high. Assign units to coders at random or by a fixed rule, and be wary if one coder’s gaps cluster on one kind of unit.
Nominal, ordinal, interval or ratio?
The level of measurement decides how much a disagreement costs. Under nominal, 4 against 5 is as wrong as 1 against 5. Under ordinal and interval it is a near miss.
Load “Severity ratings”: four reviewers rate 20 usability problems from 1 to 5, usually within one point of each other. Exact matches are only 46 % of pairs, so nominal alpha is 0.34, while ordinal alpha is 0.86. Both are correct. The first asks whether reviewers pick the identical number, and the second whether they see the same severity. For a severity scale the second is the question you mean.
| Level | δ² | Use for |
|---|---|---|
| Nominal | 0 if c = k, otherwise 1 | Themes, tags, yes or no |
| Ordinal | (Σ n_g from c to k − (n_c + n_k)/2)² | Severity, Likert items, ranks |
| Interval | (c − k)² | Scores with equal steps |
| Ratio | ((c − k)/(c + k))² | Counts, durations, money |
Note that the ordinal difference depends on how often the values between c and k were used, not on their numeric labels. Decide the level from the scale itself. Trying all four and reporting the highest is the one thing not to do with the table on this page, which is there so you can see how much the choice matters.
The confidence interval
The formula gives no standard error, so the interval comes from resampling. This page resamples whole units with replacement up to 1,000 times, computes alpha each time and takes the percentiles. Zapf and colleagues (2016) use the same approach for alpha.
The KALPHA macro by Hayes and Krippendorff (2007) resamples differently, so its limits can differ a little from these. The point estimate is the same.
How far to trust it. In our own simulation (four categories, three coders, true alpha 0.80), the nominal 95 % interval covered the true value about 93 % of the time with 20 or 30 units and 95 % with 100. At interval level, with 30 units, it was about 91–92 %. So it runs slightly narrow on small samples, and its limits are approximate there.
The KPI next to it gives the share of resamples below 0.80, with 0.667 beneath it. It is a compact way to state the risk of falling short of a threshold, not an exact probability.
Krippendorff’s alpha vs Cohen’s kappa and Fleiss’ kappa
| Situation | Use |
|---|---|
| Two raters, no gaps, and you want to report the conventional figure | Cohen’s kappa |
| Three or more raters, no gaps, unordered labels | Fleiss’ kappa, or alpha |
| Any gaps you want to keep | Alpha |
| Ordinal, interval or ratio data with several coders | Alpha |
| Numeric scores, and you need to separate rater level from noise | Intraclass correlation |
| Items of one questionnaire rather than coders | Cronbach’s alpha, a different statistic despite the name |
How close are the numbers? Chance is estimated from all the values pooled, as in Scott’s pi and Fleiss’ kappa, not from each rater’s own distribution as in Cohen’s. With no gaps and nominal data the link is exact: 1 − α = (1 − κ) · (n − 1)/n, where κ is Fleiss’ kappa and n the number of values. So on any real sample they almost coincide, and alpha’s correction matters only when the sample is tiny.
Same paradox, too. When one category dominates, alpha, like kappa, comes out low next to high agreement. That is not a fault of the formula: if nearly everything gets one label, matching on it shows little ability to tell units apart.
Calculating Krippendorff’s alpha in Excel, SPSS, R and Python
Excel has no alpha function. You can build it from the formula: count each value per unit with COUNTIF, build the coincidence matrix with SUMPRODUCT weighted by 1/(m − 1), then apply the difference function. It works for a fixed table, but it is easy to get wrong when a gap changes m, and there is no interval. Pasting the sheet here gives the result and all four levels.
SPSS and SAS: Hayes’s KALPHA macro, which also gives bootstrap limits. R: irr::kripp.alpha() takes a matrix with raters in rows, and the krippendorffsalpha package adds intervals. Python: the krippendorff package has krippendorff.alpha(), also with coders in rows by default. When a result looks wrong, check the layout first; “Coders in rows” above handles it here.
Where this comes up in product and UX research
Any time people turn open text into categories: tagging support tickets, coding interview transcripts, sorting feedback into themes, or rating the severity of usability problems. A count like “40 % of comments are about usability” only means something if another person would have counted the same.
That is what “Coded comments” shows. Bug and pricing reach 1.00, while feature and usability are at 0.38 and 0.43: the coders are mixing up those two labels, as the coincidence matrix shows. The overall 0.72 hides it. The fix is a sharper definition of those two categories in the codebook, and then a second round of coding on fresh units.
Sources
- Krippendorff K. “Computing Krippendorff’s Alpha-Reliability.” University of Pennsylvania, 2011 (literature updated 2013).
- Krippendorff K. Content Analysis: An Introduction to Its Methodology, 3rd edition. Sage, 2013, chapter 12.
- Krippendorff K. “Reliability in content analysis: some common misconceptions and recommendations.” Human Communication Research 30(3), 2004, 411–433.
- Hayes A. F., Krippendorff K. “Answering the call for a standard reliability measure for coding data.” Communication Methods and Measures 1(1), 2007, 77–89.
- Zapf A., Castell S., Morawietz L., Karch A. “Measuring inter-rater reliability for nominal data: which coefficients and confidence intervals are appropriate?” BMC Medical Research Methodology 16, 2016, 93.
FAQ
What is Krippendorff’s alpha?
A reliability coefficient for how well two or more coders agree beyond chance. It works for any number of coders, for nominal, ordinal, interval and ratio data, and with missing ratings. 1 is perfect agreement and 0 is what chance would produce.
What is a good Krippendorff’s alpha?
Krippendorff’s convention is to rely on 0.800 and above, and to use 0.667 to 0.800 only for tentative conclusions. Below 0.667, do not rely on the variable. Check the lower limit of the interval, not only the point, and set the bar by what a wrong conclusion would cost.
What is the difference between Krippendorff’s alpha and Cohen’s kappa?
Kappa needs complete data and, in Cohen’s form, two raters. Alpha takes any number of coders, keeps units with gaps and handles ordered and numeric data. On complete nominal data alpha sits very close to Fleiss’ kappa.
How does it handle missing data?
A unit contributes whatever pairs of ratings it has. Only a unit with fewer than two ratings drops out. The result is valid as long as the gaps have nothing to do with the content of the units.
Which level of measurement should I choose?
Follow the scale. Nominal for unordered labels, ordinal for ranks such as severity or Likert items, interval for scores with equal steps, and ratio for quantities with a true zero. Choose before looking at the results.
Can Krippendorff’s alpha be negative?
Yes. A negative value means coders disagree more than chance would produce, which points to something systematic, such as two readings of the codebook or a reversed scale.
Can I calculate Krippendorff’s alpha in Excel?
There is no built-in function, though it can be built from the formula with a coincidence matrix. A gap in the data changes the weights, so it is easy to get wrong. Pasting the same sheet here gives all four levels with an interval.
Why does this differ from my KALPHA interval?
The point estimate should match to rounding. The interval comes from a different resampling scheme, so limits can differ a little. If you report the macro’s limits, cite the macro.
Is my data uploaded anywhere?
No. Parsing and every calculation run in your browser. Only what you put into the URL yourself, by copying the share link, leaves the page.
More in UX research
- Survey sample size calculator
How many responses does a survey need for a given margin of error?
- Kano model survey analyzer
Which features are must-haves, performance features or delighters for users?
- Van Westendorp calculator
What price range do buyers accept, and where do the price sensitivity meter curves cross?
- Gabor–Granger analysis
At which price on the ladder does revenue peak, and how elastic is demand?
- NPS, CSAT and CES calculator
What is the Net Promoter Score of these answers, with a confidence interval?
- CSAT & CES calculator
What are the CSAT and Customer Effort Score of these survey results?
- SUS calculator
What is the System Usability Scale score, and is the usability good?
- UEQ calculator
How does the product score on the six UEQ scales, and where does it sit against the benchmark?
- NASA-TLX calculator
How heavy did this task feel, and which kind of demand made it heavy?
- Cronbach’s alpha calculator
Do the items of this questionnaire scale hold together, and which item pulls the alpha down?
- Cohen’s kappa calculator
Do two or more raters agree beyond chance, and which label or rater is behind the disagreement?
- Intraclass correlation calculator
Is a numeric score a property of the subject or of the person scoring, and which ICC should you report?