Elamur Research⌕
☰
home / papers / 2014 / Learning part-of-speech taggers with inter-annotator agreement loss

Learning part-of-speech taggers with inter-annotator agreement loss

Summary

POS-annotated corpora are usually treated as fully reliable once disagreements between annotators are resolved. In practice, those disagreements cluster on linguistically debatable cases, so not every tagging error deserves the same penalty. This paper makes a Twitter POS tagger aware of human uncertainty. Two annotators tagged 500 newly sampled tweets with the universal tagset (Petrov et al., 2012), working without guidelines. From these annotations the authors estimated, for each tag, how reliably people agree, using both inter-annotator F1 scores and tag confusion probabilities. These estimates rescale the updates of a structured perceptron with drop-out. When the model confuses tags that humans also tend to confuse, the update is small. When it confuses easy tags, the update is large. The models were trained on RITTER and GIMPEL data mapped to universal tags and tested on FOSTER and a new in-house set. The cost-sensitive taggers performed better across annotation projects and, unexpectedly, also on data that follows the same guidelines. They improved downstream chunking, while NER results were mixed. The main lesson is that annotator confidence should shape learning instead of being hidden by adjudication.

Key points

  • Agreement was estimated on 500 doubly-annotated tweets, with raw agreement of 0.88 and Cohen's κ of 0.84. Agreement was high for nouns, verbs and punctuation, but low for particles, numerals and X.
  • There are two loss variants. The F1-weighted variant scales the loss by the inter-annotator F1 of the gold tag. The CM-weighted variant uses one minus the confusion probability of the gold/predicted tag pair.
  • Training data combined RITTER-TRAIN (Derczynski et al. splits; about 16k tokens overall) and GIMPEL-TRAIN (October splits, v0.3; about 26k tokens). All corpora were mapped to the 12 universal tags, and URLs, usernames and hashtags were treated as NOUN.
  • A new test set, IN-HOUSE (lowlands.test), contains 200 tweets (3,064 tokens) annotated directly in universal tags. Its first-round agreement was raw 0.9, κ 0.87 and Krippendorff's α 0.87. The set and the agreement data are publicly released on Bitbucket.
  • Downstream evaluation covered chunking and NER, both of which use POS tags as features. Chunking benefited, and NER showed mixed outcomes.

Interesting ideas

  • I1. Turning agreement statistics into per-label costs. A small doubly-annotated sample can supply weights for the training loss. Measured agreement then becomes a training signal rather than a figure that only appears in the paper. This could transfer to any annotation task where some labels are notoriously ambiguous, such as NER types, sentiment or dependency relations.
  • I2. Pair-aware penalties from a confusion matrix. Weighting errors by how often humans confuse that specific gold/predicted pair is more expressive than a per-tag weight. It means "plausible" mistakes are penalized less than absurd ones. This is cheap to add to any perceptron-style or online learner.
  • I3. Collecting agreement data without guidelines. Letting annotators follow their own judgment exposes where the real linguistic uncertainty lies, rather than where a guideline happened to draw an arbitrary line. This could be a useful diagnostic before writing annotation guidelines for a new language or domain.
  • I4. Using inconsistent corpora instead of discarding them. Mapping differently annotated datasets to a shared universal tagset, while softening the penalty on contested tags, helps a model generalize across annotation projects. This is relevant wherever several mutually inconsistent resources exist for one language.
  • I5. Evaluating beyond tagging accuracy. Checking whether a tagger change helps downstream tasks like chunking and NER shows whether "softer" training actually yields more useful features, not just different accuracy numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Part-of-speech tagging

Cited by

84 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 64 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-1078

@inproceedings{plank2014learning,
  title = {{Learning part-of-speech taggers with inter-annotator agreement loss}},
  author = {Plank, Barbara and Hovy, Dirk and Søgaard, Anders},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.3115/v1/e14-1078},
  url = {https://doi.org/10.3115/v1/e14-1078}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eyal Beigman and Beata Klebanov. 2009. Learning with annotation noise. In ACL. find
  2. Jacob Cohen et al. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  3. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In EMNLP. find our card
  4. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In ACL. find our card
  5. Leon Derczynski, Alan Ritter, Sam Clark, and Kalina Bontcheva. 2013. Twitter part-of-speech tagging for all: overcoming sparse and noisy data. In RANLP. find
  6. Jakob Elming, Anders Johannsen, Sigrid Klerke, Emanuele Lapponi, Hector Martinez, and Anders Søgaard. 2013. Down-stream effects of tree-to-dependency conversions. In NAACL. find
  7. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in Twitter data with crowdsourcing. In NAACL-HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  8. Jennifer Foster, Ozlem Cetinoglu, Joachim Wagner, Josef Le Roux, Joakim Nivre, Deirde Hogan, and Josef van Genabith. 2011. From news to comments: Resources and benchmarks for parsing the language of Web 2.0. In IJCNLP. find
  9. Hege Fromheide, Dirk Hovy, and Anders Søgaard. 2014. Crowdsourcing and annotating NER for Twitter \#drift. In Proceedings of LREC 2014. find
  10. Peter Geibel and Fritz Wysotzki. 2003. Perceptron based learning with example dependent and noisy costs. In ICML. find
  11. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In ACL. find our card
  12. Shohei Higashiyama, Kazuhiro Seki, and Kuniaki Uehara. 2013. Clinical entity recognition using cost-sensitive structured perceptron for NTCIR-10 MedNLP. In NTCIR. find
  13. Jasper Wilson Holley and Joy Paul Guilford. 1964. A Note on the G-Index of Agreement. Educational and Psychological Measurement, 24(4):749. find
  14. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. When POS datasets don’t add up: Combatting sample bias. In Proceedings of LREC 2014. find
  15. Richard Johansson. 2013. Training parsers on incompatible treebanks. In NAACL. find
  16. Beata Klebanov and Eyal Beigman. 2009. From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. find
  17. Klaus Krippendorf, 2004. Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA. find
  18. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In ICML. find
  19. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing, pages 171–189. Springer. find our card
  20. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  21. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In ACL. find
  22. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In NAACL. find our card
  23. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In LREC. find
  24. Dennis Reidsma and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguistics, 34(3):319–326. find
  25. Dennis Reidsma and Rieks op den Akker. 2008. Exploiting ‘subjective’ annotations. In Workshop on Human Judgements in Computational Linguistics, COLING. find
  26. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In EMNLP. find our card
  27. Roy Schwartz, Omri Abend, Roi Reichart, and Ari Rappoport. 2011. Neutralizing linguistically problematic annotations in unsupervised dependency parsing evaluation. In ACL. find
  28. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In NAACL. find
  29. Anders Søgaard. 2013a. Part-of-speech tagging with antagonistic adversaries. In ACL. find
  30. Anders Søgaard. 2013b. Zipfian corruptions for robust pos tagging. In NAACL. find
  31. Hyun-Je Song, Jeong-Woo Son, Tae-Gil Noh, Seong-Bae Park, and Sang-Jo Lee. 2012. A cost sensitive part-of-speech tagging: differentiating serious errors from minor errors. In ACL. find
  32. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2012. Cross-framework evaluation for statistical parsing. In EACL. find
  33. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In IJCNLP. find our card
  34. Daniel Zeman. 2010. Hard problems of tagset conversion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/