Elamur Research⌕
☰
home / papers / 2018 / Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Summary

Word-level neural language models usually learn a separate output vector for each word in a fixed vocabulary, so rare forms receive poorly trained parameters. The problem is worst in morphologically-rich languages (MRLs), where inflection produces many distinct forms and high type/token ratios. Gerz et al. tackle it in two ways. First, they assemble LM benchmarks for 50 languages selected for typological variety. They start from the Polyglot Wikipedia (PW), which is dominated by Indo-European and Western European languages, and add Wikipedia data for isolating, agglutinative and introflexive languages such as Burmese, Khmer, Basque, Georgian, Tamil and Amharic. Second, they extend the Char-CNN-LSTM model of Kim et al. (2016) with an unsupervised fine-tuning step (+AP). After every epoch, this step adjusts the output word embeddings using word pairs sampled from character-level convolutional filter features, so the output vectors also encode subword similarity. In a full-vocabulary setup, perplexity falls for 47 of the 50 languages, with the largest drops for MRLs. The result is a broad multilingual testbed and evidence that output-side subword information matters.

Key points

  • The PW starting set covers 40 languages, of which 26 are Indo-European and 28 are Western European. The authors added under-represented morphological types to reach 50 languages, and they retokenised Chinese, Japanese and Thai with the Polyglot tokeniser.
  • Reported perplexity reductions on the authors' benchmarks include about 22 points for English, 144 for Hebrew, 378 for Finnish and 957 for Korean.
  • The gains carry over to the larger MWC dataset of Kawakami et al. (2017), which covers 7 languages, and to a Europarl subset. Improvements track the corpus type/token ratio rather than its size, and Europarl has lower ratios than Wikipedia.
  • In a controlled German Wikipedia experiment, the authors used 25K sentences split 20K/2.5K/2.5K and varied the type/token ratio from 0.13 to 0.18. The benefit of +AP grew with the ratio: perplexity went from 225 to 217 at 0.13 and from 681 to 598 at 0.18.
  • Code and data are released at people.ds.cam.ac.uk/dsg40/lmmrl.html. Handling words unseen at test time (open vocabulary) is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Language models & pretraining Morphology

 Institutions

🎓University of CambridgeCambridge · United Kingdom119 works · 110,950 citations🎓Johns Hopkins UniversityBaltimore · United States151 works · 84,111 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel20 works · 3,844 citations

Cite

Cite as Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart and Anna Korhonen. 2018. Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00032

@article{gerz2018language,
  title = {{Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction}},
  author = {Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart and Anna Korhonen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00032},
  url = {https://doi.org/10.1162/tacl_a_00032}
}

References

51 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man, Rajat Mon… our card
  2. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of EACL, pages 937–947. find
  3. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183–192. our card
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. our card
  5. Balthasar Bickel and Johanna Nichols, 2013. Inflectional Synthesis of the Verb. Max Planck Institute for Evolutionary Anthropology, Leipzig. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia… our card
  6. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In Proceedings of ICML, pages 1899–1907. find
  7. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In Proceedings of BMVC. find
  8. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2014. One billion word benchmark for measuring progress in statistical language modeling. In Proceedings of INTER-SPEECH, pages 2635–2639. our card
  9. Xie Chen, Xunying Liu, Yanmin Qian, M.J.F. Gales, and Philip C Woodland. 2016. CUED-RNNLM: An open-source toolkit for efficient training and evaluation of recurrent neural network language models. In Proceedings of ICASSP, pages 6000 –6004. find
  10. Ryan Cotterell, Hinrich Schütze, and Jason Eisner. 2016. Morphological smoothing and extrapolation of word embeddings. In Proceedings of ACL, pages 1651–1660. find
  11. Koen Deschacht and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the latent words language model. In Proceedings of EMNLP, pages 21– 29. find
  12. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  13. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H Hovy, and Noah A Smith. 2015. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of NAACL-HLT, pages 1606–1615. find
  14. Christiane Fellbaum. 1998. find
  15. WordNet: An Electronic Lexical Database. Bradford Books. Katja Filippova, Enrique Alfonseca, Carlos A. Col-menares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In Proceedings of EMNLP, pages 360–368. our card
  16. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT, pages 758– 764. our card
  17. Joshua T. Goodman. 2001. A bit of progress in language modeling. Computer Speech & Language, 15(4):403– 434. find
  18. Edouard Grave, Moustapha Cissé, and Armand Joulin. 2017. Unbounded cache model for online language modeling with open vocabulary. In Proceedings of NIPS, pages 6044–6054. find
  19. Alex Graves. 2013. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850. find
  20. Derek Greene and Padraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of ICML, pages 377–384. find
  21. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of ACL, pages 690–696. our card
  22. Harold Stanley Heaps. 1978. Information retrieval, computational and theoretical aspects. find
  23. Academic Press. Gustav Herdan. 1960. Type-token mathematics, volume 4. find
  24. Mouton. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735– 1780. find
  25. Marcus Hutter. 2012. The human knowledge compression contest. find
  26. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. In Proceedings of ICML. find
  27. Dan Jurafsky and James H. Martin. 2017. Speech and Language Processing, volume 3. Pearson. find
  28. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of ACL, pages 1492–1502. find
  29. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of AAAI, pages 2741–2749. our card
  30. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proceedings of ICASSP, pages 181–184. find
  31. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86. find
  32. Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Handwritten digit recognition with a back-propagation network. In Proceedings of NIPS, pages 396–404. find
  33. Wang Ling, Tiago Luís, Luís Marujo, Ramón Fernández Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of EMNLP, pages 1520–1530. our card
  34. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of ACL, pages 1054–1063. our card
  35. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL, pages 142–150. our card
  36. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network base… our card
  37. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In Proceedings of EMNLP, pages 1992–1997. Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialisation of distributio… find
  38. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of ICML, pages 807–814. find
  39. Nelleke Oostdijk. 2000. The spoken Dutch corpus. Overview and first evaluation. In Proceedings of LREC, pages 887–894. find
  40. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of EACL, pages 157–163. our card
  41. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of AAAI, pages 3776–3784. our card
  42. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. In Proceedings of the ICML Deep Learning Workshop. our card
  43. Martin Sundermeyer, Hermann Ney, and Ralf Schluter. 2015. From feedforward to recurrent LSTM neural networks for language modeling. IEEE Transactions on Audio, Speech and Language Processing, 23(3):517– 529. find
  44. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of LREC, pages 2214–2218. find
  45. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of ACL, pages 2016–2027. find
  46. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In Proceedings of EMNLP, pages 1387–1392. Lyan Verwimp, Joris Pelemans, Hugo Van hamme, and Patrick Wambacq. 2017. Character-word LSTM language models. In Pr… our card
  47. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In Proceedings of ACL, pages 1319–1329. find
  48. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL, 3:345–358. our card
  49. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In Proceedings of ICLR. our card
  50. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of ECCV, pages 818–833. our card
  51. George Kingsley Zipf. 1949. Human behavior and the principle of least effort: An introduction to human ecology. Martino Fine Books. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gerz-2018-language-modeling-morphologically-rich-languages/