Elamur Research⌕
☰
home / papers / 2013 / Learning a Part-of-Speech Tagger from Two Hours of Annotation

Learning a Part-of-Speech Tagger from Two Hours of Annotation

Paper · 2013 · Dan Garrette and Jason Baldridge

Summary

The paper asks how well a part-of-speech tagger can be built when a language has almost no annotated data. Earlier weakly supervised work often relied on tag dictionaries taken from large treebanks, which were sometimes pruned by frequency or even included test data. To test a more honest setting, the authors gave linguists two hours each to annotate Kinyarwanda and Malagasy, with English as a control. Each linguist produced two resources: a tag dictionary for the most frequent word types (type supervision) and a set of tagged sentences (token supervision). The method has four stages. Label propagation (Modified Adsorption) over a token/feature graph first spreads the sparse annotations across a 100k-token raw corpus. A weighted model minimization step then estimates tag-sequence and word-tag frequencies. These feed EM training of an HMM, followed by a MEMM. The resulting taggers reach roughly 71–78% accuracy, compared with 63–74% without expansion and minimization, and accuracy on unknown words improves substantially. This makes realistic tagging for under-resourced languages more attainable.

Key points

  • Each annotation task had a two-hour limit and was done by a single non-native linguistics graduate student. This yielded only about 1k–2k labeled tokens or types per language.
  • The data and tag sets are:
    • English: Penn Treebank, 45 tags.
    • Kinyarwanda: genocide survivor testimonies from the Kigali Genocide Memorial Center, 14 tags.
    • Malagasy: news and Malagasy Global Voices articles, 24 tags.
    • The public-domain data is released at github.com/dhgarrette/low-resource-pos-tagging-2013.
  • The pipeline runs in four steps:
    • tag dictionary expansion via label propagation (MAD, using the Junto implementation);
    • a weighted variant of the Ravi and Knight (2009) model minimization;
    • EM-HMM training;
    • MEMM training.
  • External dictionaries with unrelated tag sets were optionally added as graph features: English Wiktionary (614k entries), malagasyworld.org (78k), and kinyarwanda.net (3.7k).
  • Accuracy on unknown words rose from 37–58% to 53–70%. With expansion and minimization, type supervision matched or beat token supervision; for Kinyarwanda it reached 79% versus 71% for token supervision.

Interesting ideas

  • I1. Time-boxed annotation as the experimental budget. Fixing annotator time, rather than corpus size, gives a much more realistic picture of what works for low-resource languages. It could be reused to compare annotation strategies for any new language or task.
  • I2. A graph built around shared feature nodes. Tokens connect to each other only through context-word and prefix/suffix nodes (affixes up to length 5), and each edge is weighted by 1/N for a feature shared by N nodes. This is a cheap, language-agnostic way to spread a handful of labels over a large raw corpus. It may suit morphologically rich languages especially well.
  • I3. Mismatched dictionaries as similarity signals. An existing dictionary whose tag set differs from the target one can still serve as features indicating which words are related. Many languages have some dictionary but no tagged corpus, so this lets those resources contribute.
  • I4. Collecting both type and token annotation from the same people. Having each annotator produce both a dictionary and tagged sentences allows a direct test of which form of annotation is more valuable. It also exposes how individual annotators differ in speed and quality.
  • I5. Model minimization as a noise filter. Choosing the smallest set of tag bigrams that explains the corpus removed wrong tags that the annotator had added and that label propagation had spread. It also inferred frequency information without any counts being supplied, which is useful whenever seed labels are noisy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Part-of-speech tagging · 91 works

Cite

Cite as Dan Garrette and Jason Baldridge. 2013. Learning a Part-of-Speech Tagger from Two Hours of Annotation. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{garrette2013learning,
  title = {{Learning a Part-of-Speech Tagger from Two Hours of Annotation}},
  author = {Dan Garrette and Jason Baldridge},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1014/}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michele Banko and Robert C. Moore. 2004. Part-of-speech tagging in context. In Proceedings of COLING, Geneva, Switzerland. find
  2. Stanley F. Chen and Joshua Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL, Santa Cruz, California, USA. our card
  3. Silviu Cucerzan and David Yarowsky. 2002. Bootstrapping a multilingual part-of-speech tagger in one person-day. In Proceedings of CoNLL, Taipei, Taiwan. find
  4. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of ACL-HLT, Portland, Oregon, USA. our card
  5. Weiwei Ding. 2011. Weakly supervised part-of-speech tagging for Chinese using label propagation. Master’s thesis, University of Texas at Austin. find
  6. Dan Garrette and Jason Baldridge. 2012. Type-supervised hidden Markov models for part-of-speech tagging with incomplete tag dictionaries. In Proceedings of EMNLP, Jeju, Korea. find
  7. Yoav Goldberg, Meni Adler, and Michael Elhadad. 2008. EM can find pretty good HMM POS-taggers (when given a good start). In Proceedings ACL. find
  8. Kazi Saidul Hasan and Vincent Ng. 2009. Weakly supervised part-of-speech tagging for morphologically-rich, resource-scarce languages. In Proceedings of EACL, Athens, Greece. find
  9. Julian Kupiec. 1992. Robust part-of-speech tagging using a hidden Markov model. Computer Speech & Language, 6(3). find
  10. Shen Li, João Graça, and Ben Taskar. 2012. Wiki-ly supervised part-of-speech tagging. In Proceedings of EMNLP, Jeju Island, Korea. find
  11. Christopher D. Manning. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In Proceedings of CICLing. our card
  12. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2). our card
  13. Bernard Merialdo. 1994. Tagging English text with a probabilistic model. Computational Linguistics, 20(2). our card
  14. Sujith Ravi and Kevin Knight. 2009. Minimized models for unsupervised part-of-speech tagging. In Proceedings of ACL-AFNLP. find
  15. Sujith Ravi, Ashish Vaswani, Kevin Knight, and David Chiang. 2010. Fast, greedy model minimization for unsupervised tagging. In Proceedings of COLING. find
  16. Amarnag Subramanya, Slav Petrov, and Fernando Pereira. 2010. Efficient graph-based semi-supervised learning of structured tagging models. In Proceedings EMNLP, Cambridge, MA. find
  17. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan Mc-Donald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. In Transactions of the ACL. Association for Computational Linguistics. find
  18. Partha Pratim Talukdar and Koby Crammer. 2009. New regularized algorithms for transductive learning. In Proceedings of ECML-PKDD, Bled, Slovenia. find
  19. Kristina Toutanova and Mark Johnson. 2008. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In Proceedings of NIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/garrette-2013-learning-part-speech-tagger-two/