Elamur Research⌕
☰
home / papers / 2008 / Survey Article: Inter-Coder Agreement for Computational Linguistics

Survey Article: Inter-Coder Agreement for Computational Linguistics

Summary

Annotated corpora are only trustworthy if independent annotators label items consistently. In computational linguistics, the K coefficient became the default way to check this after Carletta (1996), but its assumptions were often misunderstood. Artstein and Poesio survey agreement measures used in CL and content analysis. They introduce a single notation for items, categories and coders, then explain the mathematics behind Scott's π, Cohen's κ, K (Siegel and Castellan) and Krippendorff's α. They also review a decade of CL annotation practice, including discourse, dialogue acts, anaphora and word senses. They show why raw percent agreement can mislead, since chance agreement depends on how many categories there are and how unevenly items fall into them. Coder bias matters less as the number of annotators grows. Weighted, α-style coefficients suit tasks whose labels are not equally distinct, although the chosen weights change the resulting score. The authors favour 0.8 as a cautious reliability threshold but argue against a rigid cutoff. The article remains a standard methodological reference for anyone planning an annotation project.

Key points

  • Terminology: the "kappa" used in CL (K from Siegel and Castellan) is actually Fleiss's multi-coder extension of Scott's π, not Cohen's κ. The difference is whether chance is estimated from pooled or from per-coder label distributions.
  • Why percent agreement misleads:
    • In a toy dialogue-act example, observed agreement is 0.7.
    • Uniform random coding already yields 1/2 agreement with two categories and 1/3 with three.
    • With a 95%/5% category split, chance alone gives about 90.5%, so 90% observed agreement is worse than chance.
  • Choice of coefficient: K and α dominate CL practice. Krippendorff's α is recommended for hierarchical tagsets and set-valued interpretations such as anaphora. Even major efforts like the Penn Discourse Treebank and OntoNotes reported only percent agreement.
  • Thresholds: the authors' experience suggests values above 0.8 indicate reasonable quality, while some discourse corpora have proven useful at about 0.7. Di Eugenio and Glass showed that κ and K can land on opposite sides of the 0.67 threshold.
  • Reporting practice: studies should describe the number of coders, whether they worked independently, and whether they relied only on the manual. They should also include significance and confusion matrices (Teufel and Moens 2002 is cited as a model) and an independent task-based evaluation of the corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis

Cited by

124 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.07-034-r2

@article{artstein2008survey,
  title = {{Survey Article: Inter-Coder Agreement for Computational Linguistics}},
  author = {Artstein, Ron and Poesio, Massimo},
  journal = {Computational Linguistics},
  year = {2008},
  doi = {10.1162/coli.07-034-r2},
  url = {https://doi.org/10.1162/coli.07-034-r2}
}

References

111 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, James and Mark Core. 1997. DAMSL: Dialogue act markup in several layers. Draft contribution for the Discourse Resource Initiative, University of Rochester. Available at http://www.cs.rochester.edu/research/cisd/resources/damsl/ . ↗
  2. Artstein, Ron and Massimo Poesio. 2005. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005, pages 141–150, Edinburgh. find
  3. Artstein, Ron and Massimo Poesio. 2006. Identifying reference to abstract objects in dialogue. In brandial 2006: Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pages 56–63, Potsdam. find
  4. Atkins, Sue. 1992. Tools for computer-aided corpus lexicography: The Hector project. Acta Linguistica Hungarica, 41:5–71. find
  5. Babarczy, Anna, John Carroll, and Geoffrey Sampson. 2006. Definitional, personal, and mechanical constraints on part of speech annotation performance. Natural Language Engineering, 12(1):77–90. find
  6. Bartko, John J. and William T. Carpenter, Jr. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307–317. find
  7. Beeferman, Doug, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1–3):177–210. find
  8. Bennett, E. M., R. Alpert, and A. C. Goldstein. 1954. Communications through limited questioning. Public Opinion Quarterly, 18(3):303–308. find
  9. Bloch, Daniel A. and Helena Chmura Kraemer. 1989. 2 × 2 kappa coefficients: Measures of agreement or association. Biometrics, 45(1):269–287. find
  10. Brennan, Robert L. and Dale J. Prediger. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3):687–699. find
  11. Bruce, Rebecca and Janyce Wiebe. 1998. Word-sense distinguishability and inter-coder agreement. In Proceedings of EMNLP, pages 53–60, Granada. find
  12. Bruce, Rebecca F. and Janyce M. Wiebe. 1999. Recognizing subjectivity: A case study in manual tagging. Natural Language Engineering, 5(2):187–205. find
  13. Buitelaar, Paul. 1998. CoreLex : Systematic Polysemy and Underspecification. Ph.D. thesis, Brandeis University, Waltham, MA. find
  14. Bunt, Harry C. 2000. Dynamic interpretation and dialogue theory. In Martin M. Taylor, Françoise Néel, and Don G. Bouwhuis, editors, The Structure of Multimodal Dialogue II. John Benjamins, Amsterdam, pages 139–166. find
  15. Bunt, Harry C. 2005. A framework for dialogue act specification. In Proceedings of the Joint ISO-ACL Workshop on the Representation and Annotation of Semantic Information, Tilburg. Available at: http://let.uvt.nl/research/ti/sigsem/wg/discussionnotes4.htm. ↗
  16. Byron, Donna K. 2002. Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the ACL, pages 80–87, Philadelphia, PA. find
  17. Byrt, Ted, Janet Bishop, and John B. Carlin. 1993. Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5):423–429. find
  18. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. find our card
  19. Carletta, Jean, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1):13–32. find our card
  20. Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Jan C. J. van Kuppevelt and Ronnie W. Smith, editors, Current and New Directions in Discourse and Dialogue. Kluwer, Dordrecht, pages 85–112. find our card
  21. Cicchetti, Domenic V. and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558. find
  22. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  23. Cohen, Jacob. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. find
  24. Core, Mark G. and James F. Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, AAAI, Cambridge, MA. Available at: http://www.cs.umd.edu/ ∼ traum/CA/fpapers.html. ↗
  25. Craggs, Richard and Mary McGee Wood. 2004. A two-dimensional annotation scheme for emotion in dialogue. In Papers from the 2004 AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, pages 44–49. find
  26. Craggs, Richard and Mary McGee Wood. 2005. Evaluating discourse and dialogue coding schemes. Computational Linguistics, 31(3):289–295. find
  27. Davies, Mark and Joseph L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38(4):1047–1051. find
  28. Di Eugenio, Barbara. 2000. On the usage of Kappa to evaluate agreement on coding tasks. In Proceedings of LREC, volume 1, pages 441–444, Athens. find
  29. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95–101. find
  30. Di Eugenio, Barbara, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason. 1998. An empirical investigation of proposals in collaborative dialogues. In Proceedings of 36th Annual Meeting of the ACL, pages 325–329, Montreal. find
  31. Dice, Lee R. 1945. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302. find
  32. Donner, Allan and Michael Eliasziw. 1987. Sample size requirements for reliability studies. Statistics in Medicine, 6:441–448. find
  33. Doran, Christine, John Aberdeen, Laurie Damianos, and Lynette Hirschman. 2001. Comparing several aspects of human-computer and human-human dialogues. In Proceedings of the 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark. Available at: http://www.sigdial.org/workshops/workshop2/proce… ↗
  34. Eckert, Miriam and Michael Strube. 2000. Dialogue acts, synchronizing units, and anaphora resolution. Journal of Semantics, 17(1):51–89. find our card
  35. Feinstein, Alvan R. and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  36. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
  37. Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  38. Fleiss, Joseph L. 1975. Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31(3):651–659. find
  39. Francis, W. Nelson and Henry Kucera. 1982. Frequency Analysis of English Usage: lexicon and grammar. Houghton Mifflin, Boston, MA. find
  40. Geertzen, Jeroen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages 126–133, Sydney. find
  41. Gross, Derek, James F. Allen, and David R. Traum. 1993. The Trains 91 dialogues. TRAINS Technical Note 92-1, University of Rochester Computer Science Department, Rochester, NY. find
  42. Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204. find our card
  43. Hayes, Andrew F. and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77–89. find
  44. Hearst, Marti A. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33–64. find
  45. Hovy, Eduard, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of HLT–NAACL, Companion Volume: Short Papers, pages 57–60, New York. find
  46. Hsu, Louis M. and Ronald Field. 2003. Interrater agreement measures: Comments on kappa n , Cohen’s kappa, Scott’s π, and Aickin’s α. Understanding Statistics, 2(3):205–219. find
  47. Jaccard, Paul. 1912. The distribution of the flora in the Alpine zone. New Phytologist, 11(2):37–50. find
  48. Jekat, Susanne, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and J. Joachim Quantz. 1995. Dialogue acts in VERBMOBIL. VM-Report 65, Universität Hamburg, DFKI GmbH, Universität Erlangen, and TU Berlin. find
  49. Jurafsky, Daniel, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado at Boulder, Institute for Cognitive Science. find
  50. Kilgarriff, Adam. 1999. 95% replicability for manual word sense tagging. In Proceedings of the Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 277–278, Bergen, Norway. find
  51. Klavans, Judith L., Samuel Popper, and Rebecca Passonneau. 2003. Tackling the internet glossary glut: Automatic extraction and evaluation of genus phrases. In Proceedings of the SIGIR-2003 Workshop on the Semantic Web, Toronto. find
  52. Kowtko, Jacqueline C., Stephen D. Isard, and Gwyneth M. Doherty. 1992. Conversational games within dialogue. Research Paper HCRC/RP-31, Human Communication Research Centre, University of Edinburgh. find
  53. Krippendorff, Klaus. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61–70. find
  54. Krippendorff, Klaus. 1978. Reliability of binary attribute data. Biometrics, 34(1):142–144. Letter to the editor, with a reply by Joseph L. Fleiss. find
  55. Krippendorff, Klaus. 1980. Content Analysis: An Introduction to Its Methodology, chapter 12. Sage, Beverly Hills, CA. find
  56. Krippendorff, Klaus. 1995. On the reliability of unitizing contiguous data. Sociological Methodology, 25:47–76. find
  57. Krippendorff, Klaus. 2004a. Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA. find
  58. Krippendorff, Klaus. 2004b. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411–433. find
  59. Landis, J. Richard and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  60. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. CLAWS4: The tagging of the British National Corpus. In Proceedings of COLING 1994: The 15th International Conference on Computational Linguistics, Volume 1, pages 622–628, Kyoto. find
  61. Levin, James A. and James A. Moore. 1978. Dialogue-games: Metacommunication structures for natural language interaction. Cognitive Science, 1(4):395–420. find
  62. Manning, Christopher D. and Hinrich Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA. find
  63. Marcu, Daniel, Magdalena Romera, and Estibaliz Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In Workshop on Levels of Representation in Discourse, pages 71–78, University of Edinburgh. find
  64. Marcus, Mitchell P., Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  65. Marion, Rodger. 2004. The whole art of deduction. Unpublished manuscript. find
  66. Melamed, I. Dan and Philip Resnik. 2000. Tagger evaluation given hierarchical tagsets. Computers and the Humanities, 34(1–2):79–84. Available at: http://www.sahs/utmb.edu/PELLINORE/Intro to research/wad/wad/ home.htm. ↗
  67. Mieskes, Margot and Michael Strube. 2006. Part-of-speech tagging of transcribed speech. In Proceedings of LREC, pages 935–938, Genoa. find
  68. Mihalcea, Rada, Timothy Chklovski, and Adam Kilgarriff. 2004. The SENSEVAL-3 English lexical sample task. In Proceedings of SENSEVAL-3, pages 25–28, Barcelona. find our card
  69. Miltsakaki, Eleni, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. Annotating discourse connectives and their arguments. In Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 9–16, Boston, MA. find our card
  70. Moser, Megan G., Johanna D. Moore, and Erin Glendening. 1996. Instructions for Coding Explanations: Identifying Segments, Relations and Minimal Units. Technical Report 96-17, University of Pittsburgh, Department of Computer Science. find
  71. Navarretta, Costanza. 2000. Abstract anaphora resolution in Danish. In Proceedings of the 1st SIGdial Workshop on Discourse and Dialogue, Hong Kong, pages 56–65. find our card
  72. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT-NAACL 2004, pages 145–152, Boston, MA. find
  73. Neuendorf, Kimberly A. 2002. The Content Analysis Guidebook. Sage, Thousand Oaks, CA. find
  74. Palmer, Martha, Hoa Trang Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137–163. find
  75. Passonneau, Rebecca J. 2004. Computing reliability for coreference annotation. In Proceedings of LREC, volume 4, pages 1503–1506, Lisbon. find
  76. Passonneau, Rebecca J. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of LREC, Genoa, pages 831–836. find our card
  77. Passonneau, Rebecca J., Nizar Habash, and Owen Rambow. 2006. Inter-annotator agreement on a multilingual semantic annotation task. In Proceedings of LREC, Genoa, pages 1951–1956. find
  78. Passonneau, Rebecca J. and Diane J. Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of 31st Annual Meeting of the ACL, pages 148–155, Columbus, OH. find
  79. Passonneau, Rebecca J. and Diane J. Litman. 1996. Empirical analysis of three dimensions of spoken discourse: Segmentation, coherence and linguistic devices. In Eduard H. Hovy and Donia R. Scott, editors, Computational and Conversational Discourse: Burning Issues – An Interdisciplinary Account, volu… find
  80. Passonneau, Rebecca J. and Diane J. Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1):103–139. find
  81. Pevzner, Lev and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19–36. find our card
  82. Poesio, Massimo. 2004a. Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation, pages 72–79, Barcelona. find
  83. Poesio, Massimo. 2004b. The MATE/GNOME proposals for anaphoric annotation, revisited. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 154–162, Cambridge, MA. find our card
  84. Poesio, Massimo and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76–83, Ann Arbor, MI. find our card
  85. Poesio, Massimo and Natalia N. Modjeska. 2005. Focus, activation, and this-noun phrases: An empirical study. In António Branco, Tony McEnery, and Ruslan Mitkov, editors, Anaphora Processing, volume 263 of Current Issues in Linguistic Theory. John Benjamins, pages 429–442, Amsterdam and Philadelphia… find
  86. Poesio, Massimo, A. Patel, and Barbara Di Eugenio. 2006. Discourse structure and anaphora in tutorial dialogues: An empirical analysis of two theories of the global focus. Research in Language and Computation, 4(2–3):229–257. find
  87. Poesio, Massimo and Renata Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183–216. find our card
  88. Popescu-Belis, Andrei. 2005. Dialogue acts: One or more dimensions? Working Paper 62, ISSCO, University of Geneva. find
  89. Posner, Karen L., Paul D. Sampson, Robert A. Caplan, Richard J. Ward, and Frederick W. find
  90. Cheney. 1990. Measuring interrater reliability among multiple raters: An example of methods for nominal data. Statistics in Medicine, 9:1103–1115. find
  91. Rajaratnam, Nageswari. 1960. Reliability formulas for independent decision data when reliability data are matched. Psychometrika, 25(3):261–271. find
  92. Reidsma, Dennis and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguistics, 34(3):319–326. find
  93. Reinhart, T. 1981. Pragmatics and linguistics: An analysis of sentence topics. Philosophica, 27(1):53–93. find
  94. Reynar, Jeffrey C. 1998. Topic Segmentation: Algorithms and Applications. Ph.D. thesis, University of Pennsylvania, Philadelphia. find
  95. Ries, Klaus. 2002. Segmenting conversations by topic, initiative and style. In Anni R. Coden, Eric W. Brown, and Savitha Srinivasan, editors, Information Retrieval Techniques for Speech Applications, volume 2273 of Lecture Notes in Computer Science. Springer, Berlin, pages 51–66. find
  96. Rietveld, Toni and Roeland van Hout. 1993. Statistical Techniques for the Study of Language and Language Behaviour. Mouton de Gruyter, Berlin. find
  97. Rosenberg, Andrew and Ed Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proceedings of HLT-NAACL 2004: Short Papers, pages 77–80, Boston, MA. find
  98. Scott, William A. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321–325. find
  99. Shriberg, Elizabeth, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100, Cambridge, MA. find our card
  100. Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences, 2nd edition, chapter 9.8. McGraw-Hill, New York. find
  101. Stent, Amanda J. 2001. Dialogue Systems as Conversational Partners: Applying Conversation Acts Theory to Natural Language Generation for Task-Oriented Mixed-Initiative Spoken Dialogue. Ph.D. thesis, Department of Computer Science, University of Rochester. find
  102. Stevenson, Mark and Robert Gaizauskas. 2000. Experiments on sentence boundary detection. In Proceedings of 6th ANLP, pages 84–89, Seattle, WA. find our card
  103. Stolcke, Andreas, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. find our card
  104. Stuart, Alan. 1955. A test for homogeneity of the marginal distributions in a two-way classification. Biometrika, 42(3/4):412–416. find
  105. Teufel, Simone, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Proceedings of Ninth Conference of the EACL, pages 110–117, Bergen. find
  106. Teufel, Simone and Marc Moens. 2002. Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4):409–445. find our card
  107. Traum, David R. and Elizabeth A. Hinkelman. 1992. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3):575–599. find
  108. Vallduvı́, Enric. 1993. Information packaging: A survey. Research Paper RP-44, University of Edinburgh, HCRC. find
  109. Véronis, Jean. 1998. A study of polysemy judgments and inter-annotator agreement. In Proceedings of SENSEVAL-1, Herstmonceux Castle, England. Available at: http://www.itri.brighton.ac.uk/events/senseval/ARCHIVE/PROCEEDINGS/ . ↗
  110. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference, pages 45–52, Columbia, MD. find our card
  111. Zwick, Rebecca. 1988. Another look at interrater agreement. Psychological Bulletin, 103(3):374–378. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/