Elamur Research⌕
☰
home / papers / 2017 / Enriching Word Vectors with Subword Information

Enriching Word Vectors with Subword Information

Summary

Most popular word embedding models assign each word a separate vector and ignore the parts the word is made of. This is a problem for morphologically rich languages, where many word forms show up only seldom in the corpus or never at all. The authors from Facebook AI Research extended the skipgram model with negative sampling. In their model, a word is a bag of character n-grams with beginning and end markers, and its vector is the sum of the vectors of these n-grams. As a result, rare words share parameters with other words, and vectors can also be constructed for words absent from the training data (OOV). The models were trained on Wikipedia data for nine languages. They were evaluated on word similarity and analogy tasks, as well as on language modeling. The approach beat the skipgram and cbow baselines, and it also surpassed methods that depend on morphological analysis. At the same time, it requires no preprocessing or annotation. The authors released their C++ implementation publicly. For highly inflected languages such as Ukrainian, this is a practical way to obtain high-quality vectors.

Key points

  • Parameters: all n-grams from 3 to 6 characters long are used, plus the word itself as a separate unit. To limit memory, the n-grams are hashed with the FNV-1a function into 2·10⁶ buckets. The vector size is 300, with 5 negative samples per positive example.
  • Speed: the model trains about 1.5 times slower than the skipgram baseline, 105 thousand versus 145 thousand words per second per thread.
  • Language modeling: test perplexity is lower than that of the variant without subword information: 206 versus 237 for Russian, and 312 versus 339 for Czech. The model also outperforms CLBL and CANLM. For Romance languages the gain is smaller: about 3% for Spanish and 2% for French.
  • Qualitative analysis: the most important n-grams often coincide with morphemes. In the German Autofahrer, "auto" and "fahrer" stand out; in French verbs, endings such as ais> or ent> do.
  • OOV words: the vector of an unknown word is built from its n-grams, and these are meaningfully matched to parts of other words. For example, the suffix -ness corresponds to -ity in the pair scarceness / rarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Tokenization & segmentation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00051 · arXiv 1607.04606

@misc{bojanowski2017enriching,
  title = {{Enriching Word Vectors with Subword Information}},
  author = {Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  year = {2017},
  doi = {10.1162/tacl\_a\_00051},
  eprint = {1607.04606},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00051}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrei Alexandrescu and Katrin Kirchhoff. 2006. Factored neural language models. In Proc. find
  2. NAACL. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Proc. find our card
  3. EMNLP. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673– 721. find our card
  4. Giacomo Berardi, Andrea Esuli, and Diego Marcheggiani. 2015. Word embeddings go to Italy: a comparison of models and training datasets. find
  5. Italian Information Retrieval Workshop. Piotr Bojanowski, Armand Joulin, and Tomáš Mikolov. 2015. Alternative structures for character-level RNNs. In Proc. find
  6. ICLR. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In Proc. find
  7. ICML. Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. In Proc. find
  8. IJCAI. Grzegorz Chrupała. 2014. Normalizing tweets with edit scripts and recurrent neural embeddings. In Proc. find
  9. ACL. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proc. find
  10. ICML. Ryan Cotterell and Hinrich Schütze. 2015. Morphological word-embeddings. In Proc. find
  11. NAACL. Qing Cui, Bin Gao, Jiang Bian, Siyu Qiu, Hanjun Dai, and Tie-Yan Liu. 2015. KNET: A general framework for learning word embedding using morphological knowledge. ACM Transactions on Information Systems, 34(1):4:1–4:25. find
  12. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391– 407. find
  13. Cicero Nogueira dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proc. find our card
  14. COLING. Cicero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proc. find
  15. ICML. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proc. find
  16. WWW. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  17. Iryna Gurevych. 2005. Using the structure of a conceptual network in computing semantic relatedness. In Proc. find
  18. IJCNLP. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  19. Samer Hassan and Rada Mihalcea. 2009. Cross-lingual semantic relatedness using encyclopedic knowledge. In Proc. EMNLP. find
  20. Colette Joubarne and Diana Inkpen. 2011. Comparison of semantic similarity for different languages using the google n-gram corpus and second-order co-occurrence measures. In Proc. Canadian Conference on Artificial Intelligence. find
  21. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Proc. AAAI. Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. 2015. Multilingual reliability and “semantic” structure of continuous word spaces. find our card
  22. Proc. IWCS 2015. Angeliki Lazaridou, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2013. Compositionally derived representations of morphologically complex words in distributional semantics. In Proc. find
  23. ACL. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proc. find our card
  24. EMNLP. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2):203–208. find
  25. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proc. find our card
  26. ACL. Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proc. CoNLL. Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Černocký. 2012. Subword language modeling with neural netw… find our card
  27. Tomáš Mikolov, Kai Chen, Greg D. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find our card
  28. Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Adv. find our card
  29. NIPS. Alexander Panchenko, Dmitry Ustalov, Nikolay Arefyev, Denis Paperno, Natalia Konstantinova, Natalia Loukachevitch, and Chris Biemann. 2016. Human and machine judgements for russian semantic relatedness. In Proc. find
  30. AIST. Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word representations and morpheme representations. In Proc. find
  31. COLING. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Adv. find
  32. NIPS. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988. Neurocomputing: Foundations of research. chapter Learning Representations by Backpropagating Errors, pages 696–699. MIT Press. Haşim Sak, Murat Saraclar, and Tunga Gungör. 2010. Morphology-based and sub-word language modeli… find
  33. ICASSP. Hinrich Schütze. 1992. Dimensions of meaning. In Proc. IEEE Conference on Supercomputing. find
  34. Hinrich Schütze. 1993. Word space. In Adv. find
  35. NIPS. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. find our card
  36. ACL. Cyrus Shaoul and Chris Westbury. 2010. The Westbury lab Wikipedia corpus. find
  37. Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Proc. find
  38. NAACL. Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  39. Henning Sperr, Jan Niehues, and Alexander Waibel. 2013. Letter n-gram-based input encoding for continuous space language models. In Proc. of the Workshop on Continuous Vector Space Models and their Compositionality at ACL. find
  40. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proc. find
  41. ICML. Lukáš Svoboda and Tomáš Brychcin. 2016. New word analogy corpus for exploring embeddings of Czech words. In Proc. find
  42. CICLING. Peter D. Turney, Patrick Pantel, et al. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141– 188. find our card
  43. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. CHARAGRAM: Embedding words and sentences via character n-grams. In Proc. find
  44. EMNLP. Torsten Zesch and Iryna Gurevych. 2006. Automatically creating datasets for measures of semantic relatedness. In Proc. Workshop on Linguistic Distances. find
  45. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Adv. NIPS. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/