Elamur Research
☰
home / papers / 2013 / Better Word Representations with Recursive Neural Networks for Morphology

Better Word Representations with Recursive Neural Networks for Morphology

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W13-3512.pdf

Summary

Standard word embeddings give every surface form its own vector, ignoring links between related forms such as "distinct" and "distinctiveness". As a result, rare and morphologically complex words get weak vectors, and unseen words collapse into a generic placeholder. Luong, Socher and Manning propose morphological Recursive Neural Networks (morphoRNN), which use morphemes rather than whole words as the atomic units. Word vectors are assembled recursively by merging stem and affix vectors through a shared composition layer.

The paper presents two variants. The context-insensitive cimRNN learns to reconstruct existing pretrained embeddings (C&W, HSMN) from their morphemes. The context-sensitive csmRNN embeds this composition inside a neural language model trained on Wikipedia text, so gradients from context reach the morpheme level.

The authors also release a new Rare Word (RW) similarity dataset. On most benchmarks, csmRNN beats the original embeddings by a clear margin, and the gains are largest on rare words. The approach can build vectors for any unseen word made of known morphemes, which makes it promising for morphologically richer languages.

Key points

  • The composition step computes a parent vector as tanh(W_m[x_stem; x_affix] + b_m). It is applied recursively, so "unfortunately" is built from un + fortunate + ly.
  • On the RW dataset (Spearman ρ×100), csmRNN raised HSMN from 1.97 to 22.31 and C&W from 26.75 to 34.36.
  • On WS353, csmRNN improved C&W from 49.77 to 57.01 and HSMN from 62.58 to 64.58. MC and RG also improved.
  • SCWS* was the exception: csmRNN did not beat the C&W baseline there (48.48 vs 48.59).
  • The RW dataset and trained vectors are released at nlp.stanford.edu/~lmthang/morphoNLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

248 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 233 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{luong2013better,
  title = {{Better Word Representations with Recursive Neural Networks for Morphology}},
  author = {Luong, Thang and Socher, Richard and Manning, Christopher},
  booktitle = {Proceedings of the Seventeenth Conference on Computational Natural Language Learning},
  year = {2013},
  url = {https://aclanthology.org/W13-3512/}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrei Alexandrescu and Katrin Kirchhoff. 2006. Factored neural language models. In NAACL. find
  2. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. Alexander Clark. 2003. Combining distributional and morphological information for part of speech induction. In EACL. find
  4. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  5. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  6. R. Collobert. 2011. Deep learning for efficient discriminative parsing. In AISTATS. find
  7. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing, 4:3:1–3:34. find
  8. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: the concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  9. Michael Gasser and Chan-Do Lee. 1990. A short-term memory architecture for the learning of morphophonemic rules. In NIPS. find
  10. Michael Gasser. 1994. Acquiring receptive morphology: A connectionist model. In ACL. find
  11. X. Glorot, A. Bordes, and Y. Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML. find
  12. C. Goller and A. Küchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. IEEE Transactions on Neural Networks, 1:347–352. find
  13. E. H. Huang, R. Socher, C. D. Manning, and A. Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Annual Meeting of the Association for Computational Linguistics (ACL). find
  14. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In ACL. find our card
  15. Angeliki Lazaridou, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2013. Compositional-ly derived representations of morphologically complex words in distributional semantics. In ACL. find
  16. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In SLT. find
  17. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH. find
  18. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In ICASSP. find
  19. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In NAACL-HLT. find
  20. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  21. G.A. Miller. 1995. WordNet: A Lexical Database for English. Communications of the ACM. find our card
  22. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In ICML. find
  23. Andriy Mnih and Geoffrey Hinton. 2009. A scalable hierarchical distributed language model. In NIPS. find
  24. Frederic Morin. 2005. Hierarchical probabilistic neural network language model. AIstats’05. In AIS-TATS. find
  25. K. Plunkett and V. Marchman. 1991. U-shaped learning and frequency effects in a multi-layered perceptron: implications for child language acquisition. Cognition, 38(1):43–102. find
  26. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In CoNLL. find our card
  27. Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. 2007. Online subgradient methods for structured prediction. find
  28. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In NAACL. find our card
  29. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  30. David E. Rumelhart and James L. McClelland. 1986. On learning the past tenses of English verbs. In J. L. McClelland, D. E. Rumelhart, and PDP Research Group, editors, Parallel Distributed Processing. Volume 2: Psychological and Biological Models, pages 216–271. MIT Press. find
  31. Cyrus Shaoul and Chris Westbury. 2010. The Westbury lab wikipedia corpus. Edmonton, AB: University of Alberta. find
  32. Noah A. Smith and Jason Eisner. 2005. Contrastive estimation: training log-linear models on unlabeled data. In ACL. find
  33. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In EMNLP. find
  34. Richard Socher, Christopher Manning, and Andrew Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In NIPS*2010 Workshop on Deep Learning and Unsupervised Feature Learning. find
  35. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS. find
  36. R. Socher, Cliff C. Lin, A. Y. Ng, and C. D. Manning. 2011b. Parsing natural scenes and natural language with recursive neural networks. In ICML. find
  37. R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. 2011c. Semi-supervised recursive autoencoders for predicting sentiment distributions. In EMNLP. find our card
  38. R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In EMNLP. find
  39. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL. find our card