Back

Assessing Agreement on Classification Tasks: The Kappa Statistic

Jean Carletta

article1996Computational Linguistics2,591 citationsACL 2021 Test-of-Time Award (25 year)

Establishes the kappa statistic as a standard metric for assessing inter-annotator reliability on subjective classification tasks in computational linguistics, replacing inconsistent and uninterpretable evaluation methods.

Listen

Researchers in computational linguistics and cognitive science increasingly rely on subjective human judgments to identify discourse segments, conversational moves, and prosodic features in dialogue data. Without reliable agreement among coders, studies cannot be replicated, yet the field has no standard way to measure that agreement. Current practices use a variety of raw agreement percentages that appear reasonable on the surface but cannot be interpreted or compared because they ignore the level of agreement expected by chance.

The article evaluates these existing reliability measures and demonstrates why they fail, then proposes a single replacement drawn from content analysis. The author examines four representative studies that mark boundaries or assign categories, shows how each measure produces misleading figures when categories are uneven or numerous, and explains the kappa statistic, which subtracts chance agreement from observed agreement and yields a value between zero and one.

Kappa normalizes results across different numbers of coders and categories, produces one comparable figure for entire sets of coders, and supports clear benchmarks: values above 0.8 indicate good reliability while values between 0.67 and 0.8 support only tentative conclusions. It also reveals when agreement is too close to chance to be useful. In contrast, the measures used in the example studies either inflate agreement when one category dominates or remain silent on how much better than chance the coders performed.

These findings matter because unreliable or incomparable coding undermines claims about discourse structure and prosody that many current models depend on. Adopting kappa would let researchers judge whether a coding scheme is stable enough for its intended use, compare schemes across papers, and identify where instructions or category definitions need improvement.

The article recommends that the community replace existing practices with kappa, using a variant that compares coders only to an expert when one clearly exists. It notes two limitations: kappa still requires a sensible, consistent choice of coding units, and discourse phenomena may prove harder to code reliably than the newspaper or survey material for which kappa was originally developed. Results from the four studies cannot be re-calculated here, so the strength of the recommendation rests on the logical analysis of chance agreement rather than new empirical data.

Human Communication Research CentreUniversity of Edinburghclassificationcognitive sciencecorrelation analysiskappa statisticclassification taskspairwise agreement
arXiv: cmp-lg/9602004

No sufficiently relevant recommendations were found.

Cover for Assessing Agreement on Classification Tasks: The Kappa Statistic

Abstract

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better off as a field adopting techniques from content analysis.

Topics

classificationcognitive sciencecorrelation analysisdialoguepsycholinguisticsclassification models

Keywords

kappa statisticclassification taskspairwise agreementchance agreementcontent analysissubjective judgmentsconversational move boundariesdiscourse segments

Table of Contents

  • 1 Introduction
  • 2 Chance expected agreement
  • 3 The kappa statistic
  • 4 Expert versus naive coders
  • 5 Conclusions
  • 6 Author Note
  • References

Knowls

  1. Knowl 1 — The Kappa Statistic for Inter-Coder Agreement

    equation

    The kappa coefficient of agreement (KK) measures pairwise agreement among a set of coders classifying units into discrete nominal categories, correcting for agreement expected by chance:

    K=P(A)−P(E)1−P(E)K = \frac{P(A) - P(E)}{1 - P(E)}

    where:

    • P(A)P(A) is the observed proportion of pairwise agreements among coders across all classified units.
    • P(E)P(E) is the proportion of agreements expected purely by chance, calculated from the overall marginal category distributions.

    When the observed agreement matches chance expectation exactly, K=0K = 0. When there is complete consensus among all coders, K=1K = 1. Negative values of KK indicate agreement that is lower than would be expected by chance.

  2. Knowl 2 — Interpretation Benchmarks for the Kappa Statistic

    model/method

    In evaluating the reliability of subjective classification schemes using the kappa coefficient (KK), content analysis conventions establish the following heuristic thresholds:

    • K>0.8K > 0.8: Indicates good reliability.
    • 0.67<K<0.80.67 < K < 0.8: Indicates moderate reliability that allows tentative conclusions to be drawn.
    • K<0.7K < 0.7: Indicates that the coding scheme is insufficiently reliable to find statistical associations between coded variables.

    While statistical significance tests can determine whether KK differs from zero (random chance), significance alone does not guarantee data reliability; the magnitude of KK must be interpreted against these thresholds to determine whether the annotation data is sufficiently noise-free for downstream empirical use.

  3. Knowl 3 — Impact of Category Distribution and Cardinality on Expected Chance Agreement

    theoretical result

    Raw pairwise agreement percentages are uninterpretable and incomparable across tasks because expected chance agreement (P(E)P(E)) depends directly on the number of categories and their relative proportions:

    • For kk categories used in equal proportions by two random coders, expected chance agreement is P(E)=1kP(E) = \frac{1}{k} (e.g., P(E)=0.50P(E) = 0.50 for 2 categories, and P(E)=0.25P(E) = 0.25 for 4 categories).
    • When category distributions are skewed, P(E)P(E) increases substantially. For a two-category task where one category occurs with marginal probability p1=0.95p_1 = 0.95 and the other with p2=0.05p_2 = 0.05, chance agreement between two independent coders is:

    P(E)=p12+p22=(0.95)2+(0.05)2=0.9025+0.0025=0.905P(E) = p_1^2 + p_2^2 = (0.95)^2 + (0.05)^2 = 0.9025 + 0.0025 = 0.905

    Thus, two coders choosing categories randomly according to these marginal probabilities will exhibit a 90.5% raw pairwise agreement rate without genuine consensus.

  4. Knowl 4 — Inherent Bias in Majority-Opinion Agreement Metrics

    theoretical result

    Measuring coder reliability as the ratio of observed agreements with the majority opinion among NN coders to total possible agreements with the majority opinion suffers from systematic structural bias:

    • The metric inherently guarantees an agreement level of at least 50% (0.500.50) regardless of underlying coding consistency, because each coder's decision is compared exclusively against the category chosen by the majority for that item.
    • It implicitly assumes that the majority vote represents an infallible ground truth, masking coder confusion and failing to penalize weak or fragmented consensus across items.

  5. Knowl 5 — Limitations of the Boundary-Marked Overlap Ratio

    limitation

    In boundary annotation tasks (such as discourse segmentation), measuring agreement as the ratio of agreed boundary markers between a coder and an expert over the total number of boundaries marked by either coder has substantial methodological drawbacks:

    • It fails to evaluate agreement on the absence of boundaries (true negatives), which makes scores sensitive to boundary density and prevents comparison across datasets with differing base rates.
    • It aggregates false positives (marking a boundary where the expert did not) and false negatives (failing to mark an expert boundary) into a single combined denominator, failing to differentiate the distinct error modes of naive coders.

  6. Knowl 6 — Expert-Baseline Adaptation of the Kappa Statistic

    model/method

    When evaluating naive annotators against a single expert standard or gold-standard coding scheme (rather than symmetric pairwise agreement across all coder pairs), the kappa statistic is defined as:

    K=P(A)−P(E)1−P(E)K = \frac{P(A) - P(E)}{1 - P(E)}

    where:

    • P(A)P(A) is the proportion of instances where the naive coders agree with the expert's judgment.
    • P(E)P(E) is the expected proportion of agreements between the naive coders and the expert occurring by chance, calculated from the marginal category distributions of the naive coders and the expert.

    This adaptation measures how effectively written coding guidelines convey the expert's target schema to naive coders while controlling for chance agreement.

  7. Knowl 7 — Unitization Sensitivity of Kappa in Boundary Segmentation

    limitation

    Applying the kappa statistic to boundary segmentation tasks requires partitioning continuous discourse into explicit, discrete candidate units (e.g., prosodic phrase boundaries, clause boundaries, word boundaries, or phoneme boundaries) and treating each unit as a binary classification decision ("boundary" versus "no boundary"):

    • The choice of base segmentation unit directly dictates the total number of candidate sites, altering the proportion of non-boundary instances and thereby changing the expected chance agreement P(E)P(E).
    • Choosing overly fine-grained units (such as words or phonemes rather than clauses) inflates the proportion of negative cases, skewing P(E)P(E) closer to 1.
    • Consequently, kappa values cannot be compared across boundary annotation studies unless the studies share a common, pretheoretically justified unitization scheme.

Coverage note — None was omitted; all key analytical critiques, mathematical formulations, adaptations, and methodological limitations contributed by the paper are fully captured.

References

  1. 1.C. C. Berry. The kappa statistic. Journal of the American Medical Association, 268(18):2513, 1992.
  2. 2.Janet Cahn. An investigation into the correlation of cue phrase, unfilled pauses, and the structuring of spoken discourse. In Proceedings of the IRCS Workshop on Prosody in Natural Speech (IRCS Report 92-37), August 1992.
  3. 3.John O. Greene and Joseph N. Cappella. Cognition and talk: The relationship of semantic units to temporal patterns of fluency in spontaneous speech. Language and Speech, 29(2):141–157, 1986.
  4. 4.L. R. Goldman. The kappa statistic — in reply. Journal of the American Medical Association, 268(18):2513–4, 1992.
  5. 5.Barbara Grosz and Candace Sidner. Attentions, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204, 1986.
  6. 6.Jacqueline C. Kowtko, Stephen D. Isard, and Gwyneth M. Doherty. Conversational games within dialogue. Technical Report HCRC/RP-31, Human Communication Research Centre, University of Edinburgh, June 1992.
  7. 7.Helena Chmura Kraemer. Extension of the kappa coefficent. Biometrics, 36:207–216, 1980.
  8. 8.Klaus Krippendorff. Content Analysis: An introduction to its methodology. Sage Publications, 1980.
  9. 9.Diane Litman and Julia Hirschberg. Disambiguating cue phrases in text and speech. In Proceedings of the Thirteenth International Conference on Computational Linguistics (COLING-90), volume 2, pages 251–256, 1990.
  10. 10.Rebecca J. Passonneau and Diane J. Litman. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31st Annual Meeting of the ACL, pages 148–155, June 1993.
  11. 11.Irene Pimenta Rodrigues and Jos`e Gabriel P. Lopes. Temporal structure of discourse. In Proceedings of the Fourteenth International Conference on Computational Linguistics (COLING-92), volume 1, pages 331–337, 1992.
  12. 12.Kim Silverman, Mary Beckman, John Pitrelli, Mari Ostendorf, Colin Wightman, Patti Price, Janet Pierrehumbert, and Julia Hirschberg. TOBI: A standard for labeling English prosody. In International Conference on Speech and Language Processing (ICSLP), volume 2, pages 867–870, 1992.
  13. 13.Sidney Siegel and N. J. Castellan, Jr. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, second edition, 1988.
  14. 14.K. Soeken and P. Prescott. Issues in the use of kappa to assess reliability. Medical Care, 24:733–743, 1986.
  15. 15.Robert Philip Weber. Basic Content Analysis. Sage Publications, 1985.

Citation

MLA
Carletta, J. “Assessing Agreement on Classification Tasks: The Kappa Statistic”. Computational Linguistics, 1996, pp. 249–54, https://aclanthology.org/J96-2004/.
APA
Carletta, J. (1996). Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, 249–254. https://aclanthology.org/J96-2004/
Chicago
Carletta, J. 1996. “Assessing Agreement on Classification Tasks: The Kappa Statistic”. Computational Linguistics, 249–54. https://aclanthology.org/J96-2004/.
Harvard
Carletta, J. (1996) “Assessing Agreement on Classification Tasks: The Kappa Statistic”, Computational Linguistics. Association for Computational Linguistics, pp. 249–254. Available at: https://aclanthology.org/J96-2004/.
Vancouver
1. Carletta J (1996) Assessing Agreement on Classification Tasks: The Kappa Statistic. In: Computational Linguistics. Association for Computational Linguistics, pp 249–254

BibTeX

@article{carletta-1996-assessing,
    title = "Assessing Agreement on Classification Tasks: The Kappa Statistic",
    author = "Carletta, Jean",
    editor = "Hirschberg, Julia",
    journal = "Computational Linguistics",
    volume = "22",
    number = "2",
    year = "1996",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/J96-2004/",
    pages = "249--254"
}
Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF