Elamur Research⌕
☰
home / papers / 2017 / Morphological Inflection Generation with Hard Monotonic Attention

Morphological Inflection Generation with Hard Monotonic Attention

Summary

The paper addresses the generation of inflected word forms. The model receives a lemma (for example, the German "hart") and a set of morphosyntactic features, and must produce the required word form. Neural seq2seq models with soft attention perform well on large datasets, but on the small CELEX corpus they fall behind a specialized WFST model with latent variables. The authors propose an encoder-decoder with "hard", nearly monotonic attention. A bidirectional LSTM encodes the characters of the input word. At each step the decoder sees only one encoded character and chooses one of two options: write an output character or perform a separate step action, which moves the attention to the next position. The model does not learn the character alignment itself: it is built in advance by a separate model. Gold action sequences are derived from it, and the network is trained with standard cross-entropy. On three well-known datasets the model matches or outperforms the best neural and non-neural approaches, with the largest gains on small training sets. This matters for morphologically rich languages such as Ukrainian, which often lack annotated data.

Key points

  • The model was evaluated on three datasets: CELEX (Baayen et al., 1993), Wiktionary (Durrett & DeNero, 2013) and SIGMORPHON2016 (Cotterell et al., 2016).
  • The authors state that this is the first neural model to outperform the latent-variable WFST of Dreyer et al. (2008) on small training sets. The neural WFST of Rastogi et al. (2016) needed twice as much data to do so.
  • Training proceeds in three stages. First, hard character alignments are built with the Sudoh et al. (2013) model based on the Chinese Restaurant Process. Next, gold write/step actions are derived from them. Finally, the neural transducer is trained on these actions.
  • The model keeps a full representation of the input (Resolution Preserving): the input word is not compressed into a single fixed-size vector. Decoding time is linear in the output length, whereas with soft attention it depends on the product of the input and output lengths.
  • An analysis of the representations showed that soft attention strongly encodes character position in the encoder states. The hard model gets positional information through the step actions, which may explain its advantage on small data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

98 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1183 · arXiv 1611.01487

@inproceedings{aharoni2017morphological,
  title = {{Morphological Inflection Generation with Hard Monotonic Attention}},
  author = {Aharoni, Roee and Goldberg, Yoav},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1183},
  eprint = {1611.01487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1183}
}

References

45 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Malin Ahlberg, Markus Forsberg, and Mans Hulden. 2015. Paradigm classification in supervised learning of morphology. In NAACL HLT 2015. pages 1024– 1029. find
  2. R Harald Baayen, Richard Piepenbrock, and Rijn van H. 1993. The {CELEX} lexical data base on {CD-ROM} . find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. Proceedings of the International Conference on Learning Representations (ICLR) . find our card
  4. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. 2016. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 4945–4949. find
  5. Maximilian Bisani and Hermann Ney. 2008. Joint-sequence models for grapheme-to-phoneme conversion. Speech Commun. 50(5):434–451. https://doi.org/10.1016/j.specom.2008.01.002. — [doi:10.1016/j.specom.2008.01.002](https://doi.org/10.1016/j.specom.2008.01.002) ↗
  6. Markus Dreyer, Jason R Smith, and Jason Eisner. 2008. Latent-variable modeling of string transductions with finite-state methods. In Proceedings of the conference on empirical methods in natural language processing. pages 1080–1089. find
  7. Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In NAACL HLT 2013. pages 1185–1195. find our card
  8. Jason Eisner. 2002. Parameter estimation for probabilistic finite-state transducers. In Proceedings of the 40th annual meeting on Association for Computational Linguistics. pages 1–8. find
  9. Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In NAACL HLT 2016. find
  10. Alexander M. Fraser, Marion Weller, Aoife Cahill, and Fabienne Cap. 2012. Modeling inflection and word-formation in smt. In EACL. pages 664–674. find
  11. Mercedes Garcı́a-Martı́nez, Loı̈c Barrault, and Fethi Bougares. 2016. Factored neural machine translation. arXiv preprint arXiv:1609.04621 . — [arXiv:1609.04621](https://arxiv.org/abs/1609.04621) find
  12. Victor Chahuneau, Eva Schlinger, Noah A. Smith, and Chris Dyer. 2013. Translating into morphologically rich languages with synthetic phrases. In EMNLP. pages 1677–1687. find
  13. A. Graves and J. Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18(5-6):602–610. find
  14. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems 28, pages 577–585. find our card
  15. Alex Graves. 2012. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711 . find our card
  16. Ann Clifton and Anoop Sarkar. 2011. Combining morpheme-based machine translation with post-processing morpheme prediction. In ACL. pages 32–42. find
  17. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vy-molova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computationa… find
  18. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  19. The SIGMORPHON 2016 shared task— morphological reinflection. In Proceedings of the 2016 Meeting of SIGMORPHON. find
  20. Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a dirichlet process mixture model. In EMNLP. pages 616–627. find
  21. Mans Hulden, Markus Forsberg, and Malin Ahlberg. 2014. Semi-supervised learning of morphological paradigms and lexicons. In EACL. pages 569–578. find our card
  22. Navdeep Jaitly, David Sussillo, Quoc V Le, Oriol Vinyals, Ilya Sutskever, and Samy Bengio. 2015. A neural transducer. arXiv preprint arXiv:1511.04868 . — [arXiv:1511.04868](https://arxiv.org/abs/1511.04868) find
  23. Sittichai Jiampojamarn, Colin Cherry, and Grzegorz Kondrak. 2010. Integrating joint n-gram features into a discriminative training framework. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Com… find
  24. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099 . — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) find
  25. Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2016. Neural multi-source morphological reinflection. find
  26. Katharina Kann and Hinrich Schütze. 2016a. Med: The lmu system for the sigmorphon 2016 shared task on morphological reinflection. find
  27. Katharina Kann and Hinrich Schütze. 2016b. Single-model encoder-decoder with explicit morphological representation for reinflection. In ACL. find
  28. Ronald M. Kaplan and Martin Kay. 1994. Regular models of phonological rule systems. Computational Linguistics 20(3):331–378. find
  29. Kimmo Koskenniemi. 1983. Two-level morphology: A general computational model of word-form recognition and production. Technical report. find
  30. Liang Lu, Lingpeng Kong, Chris Dyer, Noah A Smith, and Steve Renals. 2016. Segmental recurrent neural networks for end-to-end speech recognition. arXiv preprint arXiv:1603.00223 . — [arXiv:1603.00223](https://arxiv.org/abs/1603.00223) find
  31. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1412– 1421. http… find
  32. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2283–2288. https://aclweb.org/anthology/… ↗
  33. Einat Minkov, Kristina Toutanova, and Hisami Suzuki. 2007. Generating complex morphology for machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 128– 135. http://www.a… ↗
  34. C. Raffel, T. Luong, P. J. Liu, R. J. Weiss, and D. Eck. 2017. Online and Linear-Time Attention by Enforcing Monotonic Alignments. arXiv preprint arXiv:1704.00784 . — [arXiv:1704.00784](https://arxiv.org/abs/1704.00784) find
  35. Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In Proc. of NAACL. find
  36. Carsten Schnober, Steffen Eger, Erik-Lân Do Dinh, and Iryna Gurevych. 2016. Still not there? comparing traditional sequence-to-sequence models to encoder-decoder neural networks on monotone string translation tasks. In Proceedings of COLING 2016, the 26th International Conference on Computational L… find
  37. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1703– 1714. http://aclweb.org/anthology/C16-1160. ↗
  38. Katsuhito Sudoh, Shinsuke Mori, and Masaaki Nagata. 2013. Noise-aware character alignment for bootstrapping statistical machine transliteration from bilingual corpora. In EMNLP 2013. pages 204–209. find
  39. Kristina Toutanova, Hisami Suzuki, and Achim Ruopp. 2008. Applying morphology generation models to machine translation. In ACL. pages 514–522. find
  40. David Yarowsky and Richard Wicentowski. 2000. Minimally supervised morphological analysis by multimodal alignment. In ACL. find
  41. Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1307–1316. https://aclweb.org/anthology/D16-1138. ↗
  42. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . find our card
  43. Mehryar Mohri, Fernando Pereira, and Michael Riley. 1997. A rational design for a weighted finite-state transducer library. In International Workshop on Implementing Automata. pages 144–158. find
  44. Garrett Nicolai, Colin Cherry, and Grzegorz Kondrak. 2015. Inflection generation as discriminative string transduction. In NAACL HLT 2015. pages 922–931. find
  45. Josef R. Novak, Nobuaki Minematsu, and Keikichi Hirose. 2012. WFST-based grapheme-to-phoneme conversion: Open source tools for alignment, model-building and decoding. In Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing. Association for Computatio… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/aharoni-2017-morphological-inflection-generation-hard-monotonic/