Elamur Research⌕
☰
home / papers / 2019 / Pushing the Limits of Low-Resource Morphological Inflection

Pushing the Limits of Low-Resource Morphological Inflection

Paper · 2019 · Antonios Anastasopoulos and Graham Neubig

Summary

Many of the world's languages have rich morphology but only tiny annotated datasets. Neural inflection models that excel with plenty of data break down when they get only a few dozen lemma–tag–form examples. The authors address this setting within the cross-lingual track of SIGMORPHON 2019. Their sequence-to-sequence model has separate encoders for the lemma (bidirectional RNN) and the tag set (self-attention). Its decoder attends in two steps: first over the tags, then over lemma characters using that tag-aware state. They add data hallucination, which aligns lemmas with inflected forms, treats long matching stretches as stems and replaces their interior with random letters. They also train jointly with related high-resource languages, using a schedule that avoids costly monotonic-attention constraints. Together these changes raise macro-averaged accuracy by 15 percentage points over the previous state of the art. The system ranked first on accuracy in SIGMORPHON 2019. The analysis shows transfer works best when source languages are genetically close and share a script, which is practical guidance for under-resourced languages.

Key points

  • The two-step attention decoder conditions lemma attention on the morphological tags. Its attention maps are easy to interpret, for example showing which tags produce Kazakh plural/accusative suffixes or the Greek subjunctive particle.
  • Hallucination produces 10,000 synthetic examples per low-resource language, whose original training sets can be as small as about 50 examples. Stems are defined as runs of at least 3 aligned characters, and the method also handles stem changes such as apophony.
  • A hallucination-only monolingual setup comes within a few points of combining transfer with hallucination. This makes it viable for isolates or for families without a high-resource member.
  • Extra components include coverage and Markov attention regularizers (after Cohn et al., 2016) and an adversarial language discriminator with gradient reversal. Transfer from multiple related languages beats transfer from a single one.
  • The system was best in SIGMORPHON 2019 by accuracy and third by average Levenshtein distance. It improved on Wu and Cotterell (2019), and the code is published at github.com/antonisa/inflection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Morphology · 277 worksLow-resource languages · 166 works

Cite

Cite as Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the Limits of Low-Resource Morphological Inflection. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1091

@inproceedings{anastasopoulos2019pushing,
  title = {{Pushing the Limits of Low-Resource Morphological Inflection}},
  author = {Antonios Anastasopoulos and Graham Neubig},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1091},
  url = {https://doi.org/10.18653/v1/d19-1091}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  2. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proc. CoNLL SIGMORPHON. find
  3. Judit Ács. 2018. BME-HAS system for CoNLL– SIGMORPHON 2018 shared task: Universal morphological reinflection. In Proc. CoNLL– SIGMORPHON, pages 121–126, Brussels. Association for Computational Linguistics. find
  4. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  5. The SIGMORPHON 2016 shared task— morphological reinflection. In Proc. SIGMOR-PHON. find
  6. Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In Proc. ACL.
  7. Ben Foley, Josh Arnold, Rolando Coto-Solano, Gautier Durantin, T Mark Ellison, Daan van Esch, Scott Heath, František Kratochvíl, Zara Maxwell-Smith, David Nash, et al. 2018. Building speech recognition systems for language documentation: The Co-EDL endangered language pipeline and inference system (… find
  8. Roee Aharoni, Yoav Goldberg, and Yonatan Belinkov. 2016. Improving sequence to sequence learning for morphological inflection generation: The BIU-MIT systems for the SIGMORPHON 2016 shared task for morphological reinflection. In Proc. SIGMOR-PHON. find
  9. Antonios Anastasopoulos and David Chiang. 2018. Leveraging translations for speech transcription in low-resource settings. In Proc. INTERSPEECH. find
  10. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. JMLR. find
  11. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR.
  12. Robert D Hoberman and Mark Aronoff. 2003. The verbal morphology of Maltese. Language Acquisition and Language Disorders, 28:61–78. find
  13. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. NIPS. find
  14. Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017. One-shot neural cross-lingual transfer for paradigm completion. In Proc. ACL. find
  15. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. Proc. SIGMORPHON. find
  16. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sebastian J. Mielke, Arya Mc-Carthy, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal Morphology. In Proc. LREC. find
  17. Theresa Breiner, Chieu Nguyen, Daan van Esch, and Jeremy O’Brien. 2019. Automatic keyboard layout design for low-resource Latin-script languages. arXiv:1901.06039. — [arXiv:1901.06039](https://arxiv.org/abs/1901.06039) find
  18. Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557–570.
  19. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proc. ICLR.
  20. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proc. EACL.
  21. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vy-molova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proc. NAACL-HLT.
  22. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proc. IWSLT. find
  23. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sebastian Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CoNLL–SIGMORPHON 2018 shared task: Universal morphological rein… find
  24. Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP.
  25. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proc. ICML, pages 1614–1623. find
  26. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David A Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sebastian Miel… find
  27. The SIGMORPHON 2019 shared task: Crosslinguality and context in morphology. In Proc. SIGMORPHON. find
  28. Hyperparameters Here we list all the hyperparameters of our models: • Encoder/Decoder number of layers: 1 • Character/Tag Embedding size: 32 • Recurrent State Size: 100 • LSTM Type: CoupledLSTM • Attention Size: 100 • Target and Source Character Embeddings are Tied • Tag Self-Attention size: 100 • T… find
  29. Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Miikka Silfverberg, Sebastian J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019b. find
  30. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. find
  31. Here we list the optimizer settings: Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. DyNet: The dynamic neural network toolkit. arXiv:1701.03980.
  32. Optimizer: SimpleSGD • Starting learning rate: 0.1 • Learning Rate Decay: 0.5 • Learning Rate Decay Patience: 6 epochs • Maximum number of epochs: 20, 40, 40 (for each training phase) • Minibatch Size: 10, 10, 2 (for each training phase). find
  33. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Proc. EMNLP.
  34. Ben Peters and André F. T. Martins. 2019. IT–IST at the SIGMORPHON 2019 shared task: Sparse two-headed models for inflection. In Proc. SIGMOR-PHON, Florence, Italy. find
  35. Lane Schwartz, Emily Chen, Benjamin Hunt, and Sylvia LR Schreiner. 2019. Bootstrapping a neural morphological analyzer for St. Lawrence Island Yupik from a finite-state transducer. In Proc. Comput-EL3. find
  36. Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. Proc. SIGMORPHON.
  37. James Neil Sneddon, K Alexander Adelaar, Dwi N Djenar, and Michael Ewing. 2012. Indonesian: A comprehensive grammar. Routledge. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NeurIPS.
  39. Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. arXiv:1905.06319.
  40. Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. 2017. Controllable invariance through adversarial feature learning. In Proc. NeurIPS, pages 585–596. find
  41. Chunting Zhou and Graham Neubig. 2017. Morphological inflection generation with multi-space variational encoder-decoders. In Proc. SIGMORPHON. find
  42. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In Proc. NAACL-HLT.
  43. Genetic dist. latin bengali sorani italian latvian english italian urdu slovene russian swahili portuguese kurmanji zulu kannada czech greek irish ladin lithuanian murrinhpatha neapolitan old english old saxon portuguese quechua russian sorani swahili telugu 0.86 0.88 1.0 0.30 0.25 N/A 0.10 0.88 0.8… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anastasopoulos-2019-pushing-limits-low-resource-morphological/