Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation

Summary

Some annotation tasks ask annotators to build sets rather than pick labels from a fixed inventory. Examples are grouping coreferent NPs or collecting text spans that express the same content. Standard agreement coefficients count such sets as matches only when they are identical, so near-identical annotations look unreliable. Passonneau addresses this with MASI, a set-distance measure that builds on Jaccard and gives extra credit when one set is contained in the other. MASI can serve as the weighting function inside any weighted coefficient, here Krippendorff's Alpha. The paper applies it to pyramid annotation, where Summary Content Units (SCUs) group equivalent contributors drawn from several human model summaries. Two annotators independently built pyramids for five DUC 2003 document sets. Their agreement was compared under nominal, Jaccard and MASI distances. The scores showed that SCUs rarely match exactly but overlap heavily, and that most differences are subset-like. The author argues that reliability should be judged through a "paradigmatic" study, which links agreement scores to independent evidence such as how well downstream scores correlate.

Key points

Data: five DUC 2003 Docsets, each annotated twice independently. Each pyramid used seven model summaries: four from NIST plus extra ones written by verbally skilled Columbia University recruits.
MASI was first used for coreference agreement (Passonneau, 2004) and also for the IAMTC data. It works independently of how expected agreement is estimated, so it fits Alpha or Artstein & Poesio's Beta.
Results under the three distances:
- Nominal: agreement is low, as expected.
- Jaccard: values rise several-fold.
- MASI: values close roughly half of the remaining gap to 1, which indicates mostly monotonic (subset) differences between annotators.
External validation: the paired pyramids for Docsets 30016 and 30014 were used to score 16 DUC 2003 systems. The two annotators' pyramids produced scores with Pearson correlations of 0.84–0.91 between them, which the author takes as evidence of ample reliability.
An earlier peer-annotation study on six document sets reported Kappa .57 and Alpha with Dice 0.62.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{passonneau2006measuring,
  title = {{Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation}},
  author = {Passonneau, Rebecca},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
  year = {2006},
  doi = {10.63317/4nuo6thi27ax},
  url = {https://doi.org/10.63317/4nuo6thi27ax}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Artstein, R. and M. Poesio. 2005. Kappa 3 = Alpha (or Beta). University of Essex NLE Technote 2005-01. find
Dice, L. R. (1945). Measures of the amount of ecologic association between species. Ecology 26:297-302. find
Farwell, D.; Helmreich, S.; Dorr, B. J.; Habash, N.; Reeder, F.; Miller, K.; Levin, L.; Mitamura, T.; Hovy, E.; Rambow, O.; Siddharthan, A. (2004). Interlingual annotation of multilingual text corpora. In Proceedings of the North American Chapter of the Association for Computational Linguistics Work… find
Jaccard, P. (1908). Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44:223-270. find
Krippendorff, K. 1980. Content Analysis. Newbury Park, CA: Sage Publications. find
Nenkova, A. and R. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. Proceedings of the Joint Annual Meeting of Human Language Technology (HLT) and the North American chapter of the Association for Computational Linguistics (NACL). Boston, MA. find
Passonneau, R.; Nenkova, A.; McKeown, K.; Sigelman, S. 2005. Applying the pyramid method in DUC 2005. Document Understanding Conference Workshop. find
Passonneau, R.; Habash, N.; Rambow, O. 2006. Inter-annotator agreement on a multilingual semantic annotation task. Proceedings of the International Conference on Language Resources and Evaluation (LREC). Genoa, Italy. find
Passonneau, R. 2005. Evaluating an evaluation method: The pyramid method applied to 2003 Document Understanding Conference (DUC) Data. Technical Report CUCS-010-06, Department of Computer Science, Columbia University. find
Passonneau, R. 2004. Computing reliability for coreference annotation. Proceedings of the International Conference on Language Resources and Evaluation (LREC). Portugal. find
Passonneau, R. 1997. Applying reliability metrics to coreference annotation. Technical Report CUCS-017-97, Department of Computer Science, Columbia University. find
Passonneau, R. and D. Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics 23.1: 103-139. find
Rosenberg, A. and Binkowski, E. (2004). Augmenting the kappa statistic to determine inter-annotator reliability for multiply labeled data points. In Proceedings of the Human Language Technology Conference and Meeting of the North American Chapter of the Association for Computational Linguistics (HLT… find
Siegel, S. and N. John Castellan, Jr. (1988) Non-parametric Statistics for the Behavioral Sciences, 2 nd edition. McGraw-Hill, New York. find
Teufel, S. and H. van Halteren, 2004: Evaluating information content by factoid analysis: human annotation and stability. In Proceedings of Empirical Methods in Natural Language Processing. Barcelona. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/passonneau-2006-measuring-agreement-set-valued-items/

📄 Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation