Elamur Research
☰
home / papers / 2015 / Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1508.02096

Summary

Most neural NLP systems give each word type its own independently learned vector. This ignores the shared structure of related forms, offers nothing for words unseen in training, and makes parameter counts grow with vocabulary size. These problems are especially acute for morphologically rich languages such as Turkish.

The authors propose C2W, which builds a word's embedding from its spelling. Characters are mapped to small vectors, two LSTMs read them in opposite directions, and their final states are linearly merged into a word vector. C2W takes the same input and produces the same output as a word lookup table, so it can replace one in existing architectures. Embeddings of frequent words can be cached to save computation.

The model was evaluated on language modeling and POS tagging in several languages, including the English Penn Treebank. Despite using only a fraction of the parameters, it matched or beat lookup-table baselines, with the largest gains in morphologically rich languages, and it reached a new best English PTB tagging accuracy. This suggests that much manual lexical feature engineering can be learned from characters, giving compact, open-vocabulary representations.

Key points

  • Architecture: a character lookup table feeds a bidirectional LSTM, and the word vector is a learned linear combination of the last forward and backward states. The model stores one vector per character plus fixed composition weights instead of one vector per word.
  • English PTB POS tagging accuracy:
    • Without extra features or external data, C2W scores 97.36 versus 96.70 for a word lookup table.
    • With hand-crafted features, C2W reaches 97.57.
    • C2W(tanh)+word (sskip) reaches 97.78, 0.28 points above the previous best system, SCCN (97.50).
  • As motivation, the authors note that an English Wikipedia dump of 60M sentences contains roughly 20M distinct word types after lowercasing and tokenization, and a lookup table would have to assign a separate vector to every one of them.
  • The learned embeddings place functionally similar but differently spelled words close together, such as October and January. The model can also represent words never seen in training.
  • Caching frequent words trades memory for speed. The approach was later applied to dependency parsing (Ballesteros et al., 2015), with similar gains in morphologically rich languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

185 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 168 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1508.02096 · arXiv 1508.02096

@misc{ling2015finding,
  title = {{Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation}},
  author = {Ling, Wang and Luis, Tiago and Marujo, Luis and Astudillo, Ramon Fernandez and Amir, Silvio and Dyer, Chris and Black, Alan W. and Trancoso, Isabel},
  year = {2015},
  doi = {10.48550/arxiv.1508.02096},
  eprint = {1508.02096},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1508.02096}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Susana Afonso, Eckhard Bick, Renato Haber, and Diana Santos. 2002. “Floresta sintá(c)tica”: a treebank for Portuguese. In Proc. LREC. find
  2. Nart B. Atalay, Kemal Oflazer, and Bilge Say. 2003. The annotation process in the Turkish treebank. In In Proc. the 4th International Workshop on Linguistically Interpreted Corpora (LINC). find
  3. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Proc. EMNLP. find
  4. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In Proc. ICML. find
  5. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER treebank. find
  6. Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proc. EMNLP. find our card
  7. Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. find
  8. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. EMNLP. find
  9. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR. find our card
  10. Ferdinand de Saussure. Course in General Linguistics. 1916. find
  11. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8). find
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proc. EMNLP. find
  14. Byung-Ju Kang and Key-Sun Choi. 2000. Automatic transliteration and back-transliteration by decision tree learning. In LREC. find
  15. Jiwei Li, Dan Jurafsky, and Eduard H. Hovy. 2015. When are tree structures necessary for deep learning of representations? CoRR, abs/1503.00185. find
  16. Wang Ling, João Graça, Isabel Trancoso, and Alan Black. 2012. Entropy-based pruning for phrase-based machine translation. In Proc. EMNLP. find
  17. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2013. Paraphrasing 4 microblog normalization. In Proc. EMNLP. find
  18. Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In Proc. NAACL. find our card
  19. Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A recursive recurrent neural network for statistical machine translation. In Proc. ACL. find
  20. Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In Proc. CoNLL. find our card
  21. Christopher D. Manning. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In Proc. CICLing. find our card
  22. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. Comput. Linguist. find our card
  23. William Marslen-Wilson and Lorraine K. Tyler. 1998. Rules, representations, and the English past tense. Trends in Cognitive Science, 2(11). find
  24. M. Antonia Martı́, Mariona Taulé, Lluı́s Márquez, and Manuel Bertran. 2007. CESS-ECE: A multilingual and multilevel annotated corpus. In Proc. LREC. find
  25. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proc. Interspeech. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proc. NIPS. find
  27. Robert C. Moore and Chris Quirk. 2009. Less is more: significance-based ngram selection for smaller, better language models. In Proc. EMNLP. find
  28. Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proc. EMNLP. find
  29. Tetsuji Nakagawa, Taku Kudoh, and Yuji Matsumoto. 2001. Unknown word guessing and part-of-speech tagging using support vector machines. In In Proc. the Sixth Natural Language Processing Pacific Rim Symposium. find
  30. Cicero D. Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proc. ICML. find
  31. Kristie Seymore and Ronald Rosenfeld. 1996. Scalable backoff language models. In Proc. ICSLP. find
  32. Anders Søgaard. 2011. Semisupervised condensed nearest neighbor for part-of-speech tagging. In Proc. ACL. find
  33. Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Proc. NAACL. find
  34. Drahomı́ra Spoustová, Jan Hajič, Jan Raab, and Miroslav Spousta. 2009. Semi-supervised training for the averaged perceptron POS tagger. In Proc. EACL. find
  35. Andreas Stolcke. 1998. Entropy-based pruning of backoff language models. In In Proc. DARPA Broadcast News Transcription and Understanding Workshop. find
  36. Xu Sun. 2014. Structure regularization for structured prediction: Theories and experiments. CoRR, abs/1411.6243. — [arXiv:1411.6243](https://arxiv.org/abs/1411.6243) find
  37. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Proc. Interspeech. find
  38. Wei Xu, Alan Ritter, and Ralph Grishman. 2013. Gathering and generating paraphrases from twitter with application to normalization. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora. find
  39. Richard Zens, Daisy Stanton, and Peng Xu. 2012. A systematic comparison of phrase table pruning techniques. In Proc. EMNLP. find