Elamur Research
☰
home / papers / 1996 / Assessing Agreement on Classification Tasks: The Kappa Statistic

Assessing Agreement on Classification Tasks: The Kappa Statistic

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J96-2004.pdf

Summary

Research on discourse and dialogue depends on human annotators making subjective category judgments, such as marking discourse segment boundaries or labelling conversational moves. Carletta argues that the field was defending the reliability of these annotations with an assortment of ad hoc agreement figures. These figures can't be compared with one another. None of them corrects for agreement that would happen by chance, so even a single figure can't be properly interpreted. The paper reviews four published measures: from Kowtko, Isard and Doherty; Litman and Hirschberg; Silverman et al. (TOBI prosody labelling); and Passonneau and Litman. It shows how each one inflates or hides agreement, especially when one category dominates, as it usually does in boundary marking. As a fix, it proposes the kappa coefficient, which content analysis had already adopted. Kappa adjusts observed agreement for chance, gives a single value across multiple coders and has established benchmarks for interpretation. The squib became the standard reference behind reporting kappa for annotated corpora in NLP.

Key points

  • Kappa is defined as K = (P(A) − P(E)) / (1 − P(E)), following Siegel and Castellan (1988). K = 0 means agreement is no better than chance, and K = 1 means perfect agreement.
  • Chance agreement can be large. With two categories where one is used 95% of the time, random coders would still agree 90.5% of the time. With four equally frequent categories, they would agree 25% of the time.
  • Based on Krippendorff (1980), K > .8 counts as good reliability, and values between .67 and .8 support only tentative conclusions.
  • The majority-vote "percent agreement" measure always yields at least 50% and quietly treats the majority as an expert. The paper also argues that subjective coding has no true experts. What matters is how naive coders do when given only written instructions.
  • Caveats: K values are comparable only if the coding units are chosen consistently (clause, word or phoneme boundaries give different results). When a gold-standard expert is justified, a variant of kappa can measure only how well each coder agrees with that expert.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

381 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 361 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@article{carletta1996assessing,
  title = {{Assessing Agreement on Classification Tasks: The Kappa Statistic}},
  author = {Carletta, Jean},
  journal = {Computational Linguistics},
  year = {1996},
  url = {https://aclanthology.org/J96-2004/}
}

References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berry, C. C. 1992. The kappa statistic. Journal of the American Medical Association, 268(18):2513. find
  2. Cahn, Janet. 1992. An investigation into the correlation of cue phrase, unfilled pauses, and the structuring of spoken discourse. find
  3. In Proceedings of the IRCS Workshop on Prosody in Natural Speech (IRCS Report 92-37), August. find
  4. Greene, John O. and Joseph N. Cappella. 1986. Cognition and talk: The relationship of semantic units to temporal patterns of fluency in spontaneous speech. Language and Speech, 29(2):141-157. find
  5. Goldman, L. R. 1992. The kappa statistic--in reply. Journal of the American Medical Association, 268(18):2513-4. find
  6. Grosz, Barbara and Candace Sidner. 1986. Attentions, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204. find
  7. Kowtko, Jacqueline C., Stephen D. Isard, and Gwyneth M. Doherty. 1992. Conversational games within dialogue. Technical Report HCRC/RP-31, Human Communication Research Centre, University of Edinburgh, June. find
  8. Kraemer, Helena Chmura. 1980. Extension of the kappa coefficent. Biometrics, 36:207-216. find
  9. Krippendorff, Klaus. 1980. Content Analysis: An introduction to its Methodology. Sage Publications. find
  10. Litman, Diane and Julia Hirschberg. 1990. Disambiguating cue phrases in text and speech. In Proceedings of the Thirteenth International Conference on Computational Linguistics (COLING-90), volume 2, pages 251-256. find
  11. Passonneau, Rebecca J. and Diane J. Litman. 1993. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31st Annual Meeting of the ACL, pages 148-155, June. find
  12. Rodrigues, Irene Pimenta and Jos@ Gabriel P. Lopes. 1992. Temporal structure of discourse. In Proceedings of the Fourteenth International Conference on Computational Linguistics (COLING-92), volume 1, pages 331-337. find
  13. Silverman, Kim, Mary Beckman, John Pitrelli, Marl Ostendorf, Colin Wightman, Patti Price, Janet Pierrehumbert, and Julia Hirschberg. 1992. TOBI: A standard for labeling English prosody. In International Conference on Speech and Language Processing (ICSLP), volume 2, pages 867-870. find
  14. Siegel, Sidney and N. J. Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. Second edition. McGraw-Hill. find
  15. Soeken, K. and P. Prescott. 1986. Issues in the use of kappa to assess reliability. Medical Care, 24:733-743. find
  16. Weber, Robert Philip. Basic Content Analysis. Sage Publications. find