Elamur Research⌕
☰
home / papers / 2018 / Context Sensitive Neural Lemmatization with Lematus

Context Sensitive Neural Lemmatization with Lematus

Paper · 2018 · Toms Bergmanis and Sharon Goldwater

Summary

Lemmatizers must cope with two difficulties: wordforms never seen in training, and forms whose lemma depends on the sentence (e.g., Latvian "ceļu" may come from a verb or two nouns). Earlier context-aware systems rarely measured how much context actually contributes. Bergmanis and Goldwater present Lematus, which adapts the Nematus attentional encoder-decoder to generate lemmas character by character. Its input is the target word surrounded by a window of neighbouring characters. Unlike prior systems, it needs only lemmatized text, with no POS tags, edit trees or external embeddings. The authors test it on 20 typologically diverse UD treebanks, using both full training sets and a reduced setting of 10k tokens. They compare it against a context-free variant and three baselines: Morfette, Lemming and Ch-2017. Adding context consistently improves accuracy, mostly for ambiguous forms. The context-free model is already competitive, especially on unseen words. Lematus beats Lemming with full data and matches it with 10k tokens. The study suggests that context matters most for languages with high ambiguity.

Key points

  • Data and setup: UD Treebank v2.0 standard splits for 20 languages, with Dutch taken from v2.1. Each language is tested with full data and with only the first 10k training tokens.
  • Architecture: A 2-layer bidirectional GRU encoder feeds a decoder built from a conditional GRU and a GRU, using 100 hidden units and 300-dimensional embeddings. The model uses 20 context characters on each side (Lematus 20-Ch) or none (0-Ch).
  • Effect of context (full data): Accuracy on ambiguous words rises from 88.8% to 92.4%, while unseen words improve only from 83.6% to 84.3%. Even 0-Ch outperforms all baselines on unseen words.
  • Low-resource setting: With 10k tokens, most systems lose 5.6–6.8 points. Ch-2017 loses about 10 points, and Lematus 20-Ch and Lemming are statistically equivalent.
  • Cross-lingual analysis:
    • A higher share of unseen words predicts lower accuracy (rank R = −0.73).
    • Higher ambiguity is associated with higher accuracy (R = 0.44), likely because ambiguity and productivity are anti-correlated.
    • Training-set size does not significantly predict accuracy on unseen words (R = 0.32).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Morphology · 405 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Lemma · 24 worksLemmatization · 20 works

Cite

Cite as Toms Bergmanis and Sharon Goldwater. 2018. Context Sensitive Neural Lemmatization with Lematus. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1126

@inproceedings{bergmanis2018context,
  title = {{Context Sensitive Neural Lemmatization with Lematus}},
  author = {Toms Bergmanis and Sharon Goldwater},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1126},
  url = {https://doi.org/10.18653/v1/n18-1126}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Yoav Goldberg, and Yonatan Belinkov. 2016. Improving sequence to sequence learning for morphological inflection generation: The BIU-MIT systems for the SIGMORPHON 2016 shared task for morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research i… find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.
  3. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection. Association for Computational Linguistics, Vancouver, Can… find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5:135–146.
  5. Abhisek Chakrabarty, Onkar Arun Pandit, and Utpal Garain. 2017. Context sensitive lemmatization using two successive bidirectional gated recurrent networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computation… find
  6. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In The Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computational Linguistics, Doha, Qatar.
  7. Grzegorz Chrupała. 2006. Simple data-driven context-sensitive lemmatization. Procesamiento del Lenguaje Natural 37. find
  8. Grzegorz Chrupała, Georgiana Dinu, and Josef van Genabith. 2008. Learning morphology with morfette. In Bente Maegaard Joseph Mariani Jan Odijk Stelios Piperidis Daniel Tapias Nicoletta Calzolari (Conference Chair), Khalid Choukri, editor, Proceedings of the Sixth International Conference on Language…
  9. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  10. Conll-sigmorphon 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection. Association for Computational Linguistics, Vancouver, pages 1–30.
  11. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  12. The SIGMORPHON 2016 shared taskmor-phological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 10–22. find
  13. Tomaž Erjavec and Sasčo Džeroski. 2004. Machine learning of morphosyntactic structure: Lemmatizing unknown slovene words. Applied Artificial Intelligence 18(1):17–41. find
  14. Nerea Ezeiza, Iñaki Alegria, José Marı́a Arriola, Rubén Urizar, and Itziar Aduriz. 1998. Combining stochastic and rule-based methods for disambiguation in agglutinative languages. In Proceedings of the 17th international conference on Computational linguistics-Volume 1. Association for Computatio… find
  15. Philip Gage. 1994. A new algorithm for data compression. C Users J. 12(2):23–38. find
  16. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems. pages 1019–1027.
  17. Dilek Z. Hakkani-Tür, Kemal Oflazer, and Gökhan Tür. 2002. Statistical morphological disambiguation for agglutinative languages. Computers and the Humanities 36(4):381–410. find
  18. Sittichai Jiampojamarn, Colin Cherry, and Grzegorz Kondrak. 2008. Joint processing and discriminative training for letter-to-phoneme conversion. In Proceedings of ACL-08: HLT. Association for Computational Linguistics, Columbus, Ohio, pages 905– 913. find
  19. Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of ACL. Association for Computational Linguistics. find
  20. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proc…
  21. Thomas Müller, Ryan Cotterell, Alexander Fraser, and Hinrich Schütze. 2015. Joint lemmatization and morphological tagging with lemming. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2268…
  22. Joakim Nivre et al. 2017. Universal dependencies 2.0 CoNLL 2017 shared task development and test data. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University. find
  23. Kemal Oflazer and Ìlker Kuruöz. 1994. Tagging and morphological disambiguation of turkish text. In Proceedings of the fourth conference on Applied natural language processing. Association for Computational Linguistics, pages 144–149. find
  24. Robert Östling. 2016. Morphological reinflection with convolutional neural networks. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 23–26. find
  25. Lutz Prechelt. 1998. Early stopping-but when? Neural Networks: Tricks of the trade pages 553–553. find
  26. Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR abs/1602.07868. find
  27. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. CoRR abs/1703.04357.
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1…
  29. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 2016–2027. find
  30. Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bergmanis-2018-context-sensitive-neural-lemmatization-lematus/