Elamur Research⌕
☰
home / papers / 2017 / Data Augmentation for Morphological Reinflection

Data Augmentation for Morphological Reinflection

Summary

This paper presents the CU Boulder Linguistics submission to the 2017 CoNLL-SIGMORPHON Shared Task, which focused on universal morphological reinflection. Its 52 languages, Ukrainian among them, come with training sets of between 100 and roughly 10,000 examples. Neural reinflection models do very well with abundant data but collapse when only a few hundred examples are available, because they overfit.

The authors build a character-level bidirectional LSTM Encoder-Decoder with attention, closely modeled on the 2016 winning system by Kann and Schütze. Their main contribution is in training. They create synthetic examples by locating the longest common substring of a lemma and its inflected form, treating it as an approximate stem, and replacing it in both strings with random characters. They also combine ten models through weighted voting and add a copy symbol for characters unseen in training, though the copy symbol was used in Task 2 only.

In Task 1 (inflecting a lemma from a feature set), the system clearly beats the non-neural baseline, even in the low-data setting. In Task 2 (paradigm completion), it overfits and performs poorly. The work shows that simple augmentation can make neural reinflection workable for under-resourced languages.

Key points

  • Each subtask has low, medium and high data settings. For Task 1 these are 100, 1,000 and 10,000 examples. For Task 2 they are roughly 10, 50 and 200 paradigms.
  • According to the authors, an unmodified Kann and Schütze-style model scores only about 0–1% accuracy for most languages when trained on 100 examples.
  • Accuracy in the low setting varies widely: Basque reaches 4% and Danish 68.90%. Lower scores go with more distinct feature combinations in the test data. Only 8% of Basque test feature sets appear in the low training data, compared with 99% in the high setting, where Basque reaches 100%.
  • In the low setting, the system falls below the baseline on eight languages. Most gaps are small, but Navajo trails by 7.3 points. The authors attribute this to short shared substrings between lemma and form (2.9 characters on average, versus 6.7 for Danish), which make augmentation less effective.
  • Generating replacement stems with language models instead of random characters lowered accuracy, probably because the synthetic strings resembled real training data too closely.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 46 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k17-2010

@inproceedings{silfverberg2017data,
  title = {{Data Augmentation for Morphological Reinflection}},
  author = {Silfverberg, Miikka and Wiemerslage, Adam and Liu, Ling and Mao, Lingshuang Jack},
  booktitle = {Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection},
  year = {2017},
  doi = {10.18653/v1/k17-2010},
  url = {https://doi.org/10.18653/v1/k17-2010}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Felix A. Gers and Juergen Schmidhuber. 2000. Recurrent nets that time and count. Technical report. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale . find
  2. Roee Aharoni, Yoav Goldberg, and Yonatan Belinkov. 2016. Improving sequence to sequence learning for morphological inflection generation: The BIU-MIT systems for the SIGMORPHON 2016 shared task for morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research i… find
  3. Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R. Steunebrink, and Jürgen Schmidhuber. 2015. LSTM: A search space odyssey. CoRR abs/1503.04069. find
  4. Michiel Hermans and Benjamin Schrauwen. 2013. Training and analyzing deep recurrent neural networks. In Proceedings of the 26th International Conference on Neural Information Processing Systems. Curran Associates Inc., USA, NIPS’13, pages 190–198. find
  5. Malin Ahlberg, Markus Forsberg, and Mans Hulden. 2015. Paradigm classification in supervised learning of morphology. In HLT-NAACL. find
  6. Iñaki Alegria and Izaskun Etxeberria. 2016. EHU at the SIGMORPHON 2016 shared task. a simple proposal: Grapheme-to-phoneme for inflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Ber… find
  7. Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germa… find
  8. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR 2015. find our card
  9. Lauri Karttunen and Kenneth R Beesley. 2005. Twenty-five years of finite-state morphology. Inquiries Into Words, a Festschrift for Kimmo Koskenniemi on his 60th Birthday pages 71–83. find
  10. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531 . — [arXiv:1405.3531](https://arxiv.org/abs/1405.3531) find
  11. David King. 2016. Evaluating sequence alignment for learning inflectional morphology. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germany. find
  12. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . find
  13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems. Curran Associates Inc., USA, NIPS’12, pages 1097–1105. find
  14. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  15. The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection. Association for Computational Linguistics, Vancouver, Canada. find
  16. Ling Liu and Lingshuang Jack Mao. 2016. Morphological reinflection with conditional random fields and unsupervised features. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germany. find
  17. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  18. The SIGMORPHON 2016 shared task: Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germany. find
  19. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau… find our card
  20. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. arXiv preprint arXiv:1705.00440 . find our card
  21. Garrett Nicolai, Bradley Hauer, Adam St Arnaud, and Grzegorz Kondrak. 2016. Morphological reinflection via discriminative string transduction. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics,… find
  22. Robert Östling. 2016. Morphological reinflection with convolutional neural networks. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germany. find
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. find
  24. Alexey Sorokin. 2016. Using longest common subsequence and character models to predict word forms. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Berlin, Germany. find
  25. Dima Taji, Ramy Eskander, Nizar Habash, and Owen Rambow. 2016. The Columbia University - New York University Abu Dhabi SIGMORPHON 2016 morphological reinflection shared task submission. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology.… find
  26. Ian H. Witten and Timothy C. Bell. 1991. The zero-frequency problem: estimating the probabilities of novel events in adaptive text compression. Information Theory, IEEE Transactions on (4):1085–1094. find
  27. Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. arXiv preprint arXiv:1502.01710 . — [arXiv:1502.01710](https://arxiv.org/abs/1502.01710) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/silfverberg-2017-data-augmentation-morphological-reinflection/