Clew

HomeToolsUX researchKrippendorff’s alpha calculator

Krippendorff’s alpha calculator: can you rely on this coding, gaps included?

Check intercoder reliability without the SPSS macro or a file in a set format. Paste one row per unit and one column per coder from any spreadsheet, with labels or numbers, and leave blank whatever a coder did not rate. You get alpha for the right level of measurement, a bootstrap interval, and the coder and category behind a low number.

● Free, no sign-upUpdated:

Alpha

Paste from Excel, Google Sheets or a CSV export. Leave missing ratings empty, or use NA, a dot or a dash. A header row and an id column are recognised.
Table layout
Level of measurement
Only matters for ordinal alpha on text labels. Numbers sort by themselves.
Confidence level

—

 

Alpha— 
Interval— 
Resamples below 0.80— 
Matching pairs— 
Units used— 
Pairable values— 
Alpha and its interval against Krippendorff’s thresholds
Alpha and its interval against Krippendorff’s thresholds
alpha and its bootstrap interval

Every level of measurement

Alpha under each level of measurement, with the kind of data it fits; the chosen level is highlighted
LevelFitsAlpha

Each coder

How many units each coder rated, the share of all units, and alpha with that coder left out
CoderRatedCoverageAlpha without

Each category

Pairable values in each category, the share of them matched by another coder, and alpha for that category against all the others
CategoryValuesMatchedAlpha vs rest

Coincidence matrix

Coincidences between values within units; the diagonal is agreement, and the row and column totals are the pairable values

Method. Coincidences within units, then α = 1 − Do/De, with the difference function of the chosen level (Krippendorff, 2011). A value alone in its unit cannot be paired and drops out. The interval is a percentile bootstrap: units are resampled whole, up to 1,000 times, with a fixed seed, so a table always gives the same limits. It is not the resampling of the KALPHA macro, so its limits can differ a little.

Everything is calculated in your browser — nothing you enter is sent anywhere.

How to use it

  1. Paste one column per coderCopy the block from your spreadsheet with the header row. Leave a cell empty, or write NA, where a coder did not rate that unit. An id column at the start is recognised and left out.
  2. Check the layoutRows are units by default. If each row is one coder, the layout of Krippendorff’s paper and of some R and Python examples, switch it. A verdict that makes no sense is often just this.
  3. Choose the level of measurementNominal for unordered labels, ordinal for ranks, interval for scores with equal steps, and ratio for counts and durations. Decide from the scale, not from which number looks best.
  4. Read the interval, not only the pointCompare the lower limit with the threshold you committed to. With few units, a point above 0.80 can come with an interval reaching well below 0.667.
  5. Find what is behind a low numberLook at the category table first. One weak category usually means the codebook needs a sharper definition; one coder who raises alpha when removed usually means retraining.

How Krippendorff’s alpha is calculated

Alpha compares the disagreement you observed with the disagreement you would expect if values were assigned by chance. It is 1 for perfect agreement and 0 when coders are no better than chance.

α = 1 − Do / De Do = (1 / n) · Σ Σ o_ck · δ²_ck observed De = (1 / (n(n − 1))) · Σ Σ n_c · n_k · δ²_ck expected o_ck coincidences of values c and k within units n_c how many pairable values equal c n all pairable values δ²_ck difference between c and k, set by the level of measurement

The key step is the coincidence matrix. Each unit with m ratings contributes m(m − 1) ordered pairs, and each pair counts 1/(m − 1). So every value counts once, however many coders saw that unit, and a unit with a gap simply contributes fewer pairs.

A worked example

Load “Krippendorff’s example”: 12 units and 4 observers from his own paper, with 7 of the 48 cells empty. Unit 12 has a single value, which cannot be paired, so n = 40. The same table gives 0.743 as nominal data, 0.815 as ordinal, 0.849 as interval and 0.797 as ratio, and this page reproduces all four. Note the interval: with only 11 usable units it runs from 0.40 to 1.00, so those differences between levels are well within sampling noise.

What is a good Krippendorff’s alpha?

Krippendorff’s own guidance is stricter than most kappa scales, and it has two lines rather than a ladder of adjectives.

Krippendorff’s thresholds
AlphaReading
0.800 and aboveRely on the variable
0.667 to 0.799Only for tentative conclusions
below 0.667Do not rely on it

He presents these as conventions and asks that the bar be set by what a wrong conclusion would cost, and fixed before the data arrive. A theme count in an internal research note can live with less than a coding scheme behind a published claim.

Apply the threshold to each variable. Alpha is calculated for one variable at a time, so a codebook with ten fields needs ten alphas, and averaging them hides the weak one. And look at the lower limit. Load “Coded comments”: alpha is 0.72 with an interval from 0.54 to 0.87, so this sample does not rule out either “not reliable” or “reliable”.

Missing data, and why alpha is built for it

Kappa needs every coder on every unit. In practice nobody codes everything: researchers split the load and double-code a sample, or one person leaves halfway through. Most tools then drop any row with a gap.

In “Coded comments” three people code 30 comments, and 14 comments are missing one coder. Dropping those rows would leave 16 comments and throw away 28 ratings that were actually made. Alpha uses all 76 ratings. Zapf and colleagues (2016) found that with ratings missing completely at random alpha stays stable, while running Fleiss’ kappa on the complete rows gives biased estimates.

The catch. This only works if the gaps have nothing to do with the content. If coders skip the units that are hard to code, the hard units never get a chance to disagree, and alpha comes out too high. Assign units to coders at random or by a fixed rule, and be wary if one coder’s gaps cluster on one kind of unit.

Nominal, ordinal, interval or ratio?

The level of measurement decides how much a disagreement costs. Under nominal, 4 against 5 is as wrong as 1 against 5. Under ordinal and interval it is a near miss.

Load “Severity ratings”: four reviewers rate 20 usability problems from 1 to 5, usually within one point of each other. Exact matches are only 46 % of pairs, so nominal alpha is 0.34, while ordinal alpha is 0.86. Both are correct. The first asks whether reviewers pick the identical number, and the second whether they see the same severity. For a severity scale the second is the question you mean.

The difference function of each level
Levelδ²Use for
Nominal0 if c = k, otherwise 1Themes, tags, yes or no
Ordinal(Σ n_g from c to k − (n_c + n_k)/2)²Severity, Likert items, ranks
Interval(c − k)²Scores with equal steps
Ratio((c − k)/(c + k))²Counts, durations, money

Note that the ordinal difference depends on how often the values between c and k were used, not on their numeric labels. Decide the level from the scale itself. Trying all four and reporting the highest is the one thing not to do with the table on this page, which is there so you can see how much the choice matters.

The confidence interval

The formula gives no standard error, so the interval comes from resampling. This page resamples whole units with replacement up to 1,000 times, computes alpha each time and takes the percentiles. Zapf and colleagues (2016) use the same approach for alpha.

The KALPHA macro by Hayes and Krippendorff (2007) resamples differently, so its limits can differ a little from these. The point estimate is the same.

How far to trust it. In our own simulation (four categories, three coders, true alpha 0.80), the nominal 95 % interval covered the true value about 93 % of the time with 20 or 30 units and 95 % with 100. At interval level, with 30 units, it was about 91–92 %. So it runs slightly narrow on small samples, and its limits are approximate there.

The KPI next to it gives the share of resamples below 0.80, with 0.667 beneath it. It is a compact way to state the risk of falling short of a threshold, not an exact probability.

Krippendorff’s alpha vs Cohen’s kappa and Fleiss’ kappa

Which coefficient fits
SituationUse
Two raters, no gaps, and you want to report the conventional figureCohen’s kappa
Three or more raters, no gaps, unordered labelsFleiss’ kappa, or alpha
Any gaps you want to keepAlpha
Ordinal, interval or ratio data with several codersAlpha
Numeric scores, and you need to separate rater level from noiseIntraclass correlation
Items of one questionnaire rather than codersCronbach’s alpha, a different statistic despite the name

How close are the numbers? Chance is estimated from all the values pooled, as in Scott’s pi and Fleiss’ kappa, not from each rater’s own distribution as in Cohen’s. With no gaps and nominal data the link is exact: 1 − α = (1 − κ) · (n − 1)/n, where κ is Fleiss’ kappa and n the number of values. So on any real sample they almost coincide, and alpha’s correction matters only when the sample is tiny.

Same paradox, too. When one category dominates, alpha, like kappa, comes out low next to high agreement. That is not a fault of the formula: if nearly everything gets one label, matching on it shows little ability to tell units apart.

Calculating Krippendorff’s alpha in Excel, SPSS, R and Python

Excel has no alpha function. You can build it from the formula: count each value per unit with COUNTIF, build the coincidence matrix with SUMPRODUCT weighted by 1/(m − 1), then apply the difference function. It works for a fixed table, but it is easy to get wrong when a gap changes m, and there is no interval. Pasting the sheet here gives the result and all four levels.

SPSS and SAS: Hayes’s KALPHA macro, which also gives bootstrap limits. R: irr::kripp.alpha() takes a matrix with raters in rows, and the krippendorffsalpha package adds intervals. Python: the krippendorff package has krippendorff.alpha(), also with coders in rows by default. When a result looks wrong, check the layout first; “Coders in rows” above handles it here.

Where this comes up in product and UX research

Any time people turn open text into categories: tagging support tickets, coding interview transcripts, sorting feedback into themes, or rating the severity of usability problems. A count like “40 % of comments are about usability” only means something if another person would have counted the same.

That is what “Coded comments” shows. Bug and pricing reach 1.00, while feature and usability are at 0.38 and 0.43: the coders are mixing up those two labels, as the coincidence matrix shows. The overall 0.72 hides it. The fix is a sharper definition of those two categories in the codebook, and then a second round of coding on fresh units.

Sources

  1. Krippendorff K. “Computing Krippendorff’s Alpha-Reliability.” University of Pennsylvania, 2011 (literature updated 2013).
  2. Krippendorff K. Content Analysis: An Introduction to Its Methodology, 3rd edition. Sage, 2013, chapter 12.
  3. Krippendorff K. “Reliability in content analysis: some common misconceptions and recommendations.” Human Communication Research 30(3), 2004, 411–433.
  4. Hayes A. F., Krippendorff K. “Answering the call for a standard reliability measure for coding data.” Communication Methods and Measures 1(1), 2007, 77–89.
  5. Zapf A., Castell S., Morawietz L., Karch A. “Measuring inter-rater reliability for nominal data: which coefficients and confidence intervals are appropriate?” BMC Medical Research Methodology 16, 2016, 93.

FAQ

What is Krippendorff’s alpha?

A reliability coefficient for how well two or more coders agree beyond chance. It works for any number of coders, for nominal, ordinal, interval and ratio data, and with missing ratings. 1 is perfect agreement and 0 is what chance would produce.

What is a good Krippendorff’s alpha?

Krippendorff’s convention is to rely on 0.800 and above, and to use 0.667 to 0.800 only for tentative conclusions. Below 0.667, do not rely on the variable. Check the lower limit of the interval, not only the point, and set the bar by what a wrong conclusion would cost.

What is the difference between Krippendorff’s alpha and Cohen’s kappa?

Kappa needs complete data and, in Cohen’s form, two raters. Alpha takes any number of coders, keeps units with gaps and handles ordered and numeric data. On complete nominal data alpha sits very close to Fleiss’ kappa.

How does it handle missing data?

A unit contributes whatever pairs of ratings it has. Only a unit with fewer than two ratings drops out. The result is valid as long as the gaps have nothing to do with the content of the units.

Which level of measurement should I choose?

Follow the scale. Nominal for unordered labels, ordinal for ranks such as severity or Likert items, interval for scores with equal steps, and ratio for quantities with a true zero. Choose before looking at the results.

Can Krippendorff’s alpha be negative?

Yes. A negative value means coders disagree more than chance would produce, which points to something systematic, such as two readings of the codebook or a reversed scale.

Can I calculate Krippendorff’s alpha in Excel?

There is no built-in function, though it can be built from the formula with a coincidence matrix. A gap in the data changes the weights, so it is easy to get wrong. Pasting the same sheet here gives all four levels with an interval.

Why does this differ from my KALPHA interval?

The point estimate should match to rounding. The interval comes from a different resampling scheme, so limits can differ a little. If you report the macro’s limits, cite the macro.

Is my data uploaded anywhere?

No. Parsing and every calculation run in your browser. Only what you put into the URL yourself, by copying the share link, leaves the page.

More in UX research

Open the collection →

Tours, tooltips and checklists whose impact shows up in the numbers.

Try Clew for freeHow it worksFree plan forever · no credit card

Product tours, popups and onboarding in the age of AI

7 patterns with step counts, copy rules and what to measure, what AI changes, and a checklist before you publish. PDF, 2 pages. What is inside →