Cohen's Kappa Calculator

Name: Cohen's Kappa Calculator
Author: CrossTabs

Measure inter-rater reliability and agreement

What is Cohen's Kappa?

Cohen's Kappa (κ) is a statistic that measures inter-rater agreement for categorical items. It accounts for the possibility of agreement occurring by chance, providing a more robust measure than simple percent agreement.

Formula

κ = (P_o - P_e) / (1 - P_e)

Where P_o = observed agreement and P_e = expected agreement by chance

Interpreting Kappa

Kappa	Agreement Level
< 0.00	Less than chance
0.01 - 0.20	Slight
0.21 - 0.40	Fair
0.41 - 0.60	Moderate
0.61 - 0.80	Substantial
0.81 - 1.00	Almost perfect

Guidelines from Landis & Koch (1977)

When to Use Kappa

Two raters classifying items into categories
Assessing diagnostic test reliability
Validating coding schemes
Quality control in content analysis

Weighted Kappa

For ordinal data where some disagreements are worse than others, use weighted kappa:

Linear weights: Disagreements weighted proportionally to distance
Quadratic weights: Larger disagreements weighted more heavily

Example: Medical Diagnosis

Two doctors independently classify 100 X-rays as Normal, Suspicious, or Abnormal. Cohen's Kappa measures how much better their agreement is compared to what would be expected by chance.

Try the Calculator

CrossTabs.com calculates Cohen's Kappa with:

Standard error and 95% confidence interval
Z-test for statistical significance
Linear and quadratic weighted kappa
Works with any square contingency table

Calculate Now

Step-by-Step Example

Two doctors classify 100 X-rays as "Normal" or "Abnormal":

	Doctor B: Normal	Doctor B: Abnormal	Total
Doctor A: Normal	70	5	75
Doctor A: Abnormal	10	15	25
Total	80	20	100

Step 1: Observed agreement (Pₒ) = (70 + 15) / 100 = 0.85

Step 2: Expected agreement (Pₑ) = (75×80 + 25×20) / 100² = (6000 + 500) / 10000 = 0.65

Step 3: Cohen's kappa = (Pₒ − Pₑ) / (1 − Pₑ) = (0.85 − 0.65) / (1 − 0.65) = 0.20 / 0.35 = 0.571

Interpretation: κ = 0.57 indicates moderate agreement beyond chance.

Interpreting Kappa Values

Kappa (κ)	Agreement Level
< 0.00	Less than chance (worse than random)
0.00 – 0.20	Slight agreement
0.21 – 0.40	Fair agreement
0.41 – 0.60	Moderate agreement
0.61 – 0.80	Substantial agreement
0.81 – 1.00	Almost perfect agreement

Common Mistakes to Avoid

Relying on percentage agreement alone. Two raters can agree 90% of the time by chance if one category dominates. Kappa corrects for this.
Using unweighted kappa for ordinal categories. If your categories are ordered (e.g., mild/moderate/severe), use weighted kappa instead — it gives partial credit for near-misses.
Ignoring prevalence effects. When one category is very common, kappa can be paradoxically low even with high agreement. Report prevalence alongside kappa.
Comparing kappa across studies with different prevalence. κ = 0.6 with 50/50 prevalence is not the same as κ = 0.6 with 95/5 prevalence.