Elamur Research⌕
☰
home / papers / 2014 / Linguistically debatable or just plain wrong?

Linguistically debatable or just plain wrong?

Summary

This position paper asks whether annotation projects should keep suppressing disagreement between annotators or treat it as useful signal. The authors study part-of-speech annotation using the 12 universal POS tags. Linguists without guidelines annotated English newswire (PTB WSJ), spoken transcripts (Talkbank) and Twitter. They also used crowdsourced lay annotations and French social media data. Confusion patterns between tags turn out to be strongly correlated across domains, weaker but still significant across languages, and similar for lay people. This indicates systematic ambiguity rather than random noise. Typical cases are ADP vs. PRT, ADJ vs. NOUN and PRON vs. DET. Next, the authors mined inconsistencies with variation n-grams, and experts judged whether each tag was plausible in context. Only about a fifth of disagreements were real mistakes, which is under 2% of all annotations. A classifier using surface features separated errors from hard cases only modestly. The authors conclude that most disagreements are defensible analyses. They argue that forcing agreement through guidelines biases models toward one linguistic theory.

Key points

  • English data covered three domains, with 500 doubly-annotated sentences or tweets each: WSJ Section 00, Talkbank and Twitter. Cross-domain disagreement correlations reached Spearman 0.644–0.869 and Kendall's τ 0.498–0.659.
  • In the lay study, 177 Crowdflower workers re-annotated the Gimpel et al. (2011) Twitter training set, totalling 14,619 items. Majority vote matched experts 79.54% of the time. Wiktionary filtering plus MACE raised this to 80.58%.
  • The French Social Media Bank was compared with English Twitter and showed Spearman 0.288 (p < 0.001).
  • Three PhD linguists judged 440 tag-confusion pairs from WSJ-00 and agreed 86% of the time. Only about 20% of conflicts were errors, and NOUN–PRON confusions were always hard cases.
  • An L2-regularized logistic regression model detected errors with an F1 of about 70%. Separately, nine linguistics faculty members were each shown 10 hard cases, and they disagreed on 8 of them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2083

@inproceedings{plank2014linguistically,
  title = {{Linguistically debatable or just plain wrong?}},
  author = {Plank, Barbara and Hovy, Dirk and Søgaard, Anders},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2083},
  url = {https://doi.org/10.3115/v1/p14-2083}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Technical report, Ling-Pipe. find
  2. A. Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20–28. find
  3. Markus Dickinson and Detmar Meurers. 2003. Detecting errors in part-of-speech annotation. In EACL. find
  4. Cecily Duffield, Jena Hwang, Susan Brown, Dmitriy Dligach, Sarah Vieweg, Jenny Davis, and Martha Palmer. 2007. Criteria for the manual grouping of verb senses. In LAW. find
  5. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In ACL. find our card
  6. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In NAACL. find our card
  7. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. Experiments with crowdsourced re-annotation of a POS tagging data set. In ACL. find
  8. David Juergens. 2014. An analysis of ambiguity in word sense annotations. In LREC. find
  9. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  10. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In LREC. find
  11. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In EACL. find our card
  12. Vikas C. Raykar and Shipeng Yu. 2012. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. Journal of Machine Learning Research, 13:491–518. find
  13. Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a treebank of noisy user generated content. In COLING. find
  14. Padhraic Smyth, Usama Fayyad, Mike Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of Venus images. In NIPS. find
  15. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. 2010. The multidimensional wisdom of crowds. In NIPS. find
  16. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS. find
  17. Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In AIStats. find
  18. Daniel Zeman. 2010. Hard problems of tagset conversion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plank-2014-linguistically-debatable-or-just-plain/