Elamur Research
☰
home / papers / 2017 / Advances in Pre-Training Distributed Word Representations

Advances in Pre-Training Distributed Word Representations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1712.09405

Summary

Many NLP systems start from publicly released word vectors, but training them on web-scale text is costly. The authors therefore built a stronger set of English pre-trained vectors by combining several established techniques that are rarely applied together. They extended the word2vec/fastText cbow model with three additions: position-dependent weighting of context words (following Mnih and Kavukcuoglu), multi-word phrase tokens produced by the word2phrase tool, and character n-gram subword vectors. Training data came from two sources: a mix of Wikipedia, Statmt.org News, UMBC and Gigaword (about 20B words), and a de-duplicated English Common Crawl subset (630B words). Removing duplicate sentences turned out to be a major source of quality gains.

The resulting vectors beat GloVe models trained on comparable corpora on several evaluations: word analogies, the Rare Words dataset, Squad question answering and text classification. The Wikipedia+News and Common Crawl models were published on fasttext.cc. This gives practitioners strong off-the-shelf embeddings without the expense of training on huge corpora.

Key points

  • On Common Crawl, the analogy score rises with each addition: phrases, then position weighting (window 15 versus 5 for the baseline). The authors report 87% for this setup, which subword features lift to 88.5%.
  • On the phrase-based analogy set from Mikolov et al. (2013b), the Crawl model scores 88%, which the authors describe as a new state of the art.
  • Compared with GloVe Crawl, fastText Crawl improves:
    • analogy accuracy from 75 to 85;
    • Rare Words score from 0.52 to 0.58;
    • Squad accuracy from 78.9% to 79.8%.
  • On text classification (MRPC, MR, CR, SUBJ, MPQA, SST, TREC; SentEval script by Conneau et al.), the average rises from 79.7 to 82.5 for Wiki+news models and from 82.0 to 82.7 for Crawl models.
  • Preprocessing was kept deliberately light, using only the Moses tokenizer.perl. English Common Crawl text was selected with a unigram language-model filter, which the authors note could also extract other languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

260 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 244 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1712.09405 · arXiv 1712.09405

@misc{mikolov2017advances,
  title = {{Advances in Pre-Training Distributed Word Representations}},
  author = {Mikolov, Tomas and Grave, Edouard and Bojanowski, Piotr and Puhrsch, Christian and Joulin, Armand},
  year = {2017},
  doi = {10.48550/arxiv.1712.09405},
  eprint = {1712.09405},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.09405}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baroni, M. and Lenci, A. (2010). Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  2. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  3. Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017). Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051. find our card
  4. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. find our card
  5. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. find our card
  6. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391. find
  7. Han, L., Kashyap, A. L., Finin, T., Mayfield, J., and Weese, J. (2013). UMBC EBIQUITY-CORE: Semantic Textual Similarity Systems. In Proceedings of the Second CR SUBJ MPQA SST TREC Average 78.1 79.6 91.5 91.8 86.9 88.0 78.1 80.0 66.6 84.2 79.7 82.0 80.3 81.1 92.2 92.5 81.1 82.0 82.5 82.7 88.3 87.8 85… find
  8. Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, June. find
  9. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2016). Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
  10. Levy, O., Goldberg, Y., and Dagan, I. (2015). Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225. find
  11. Li, W. (1992). Random texts exhibit zipf’s-law-like word frequency distribution. IEEE Transactions on information theory, 38(6):1842–1845. find
  12. Luong, T., Socher, R., and Manning, C. D. (2013). Better word representations with recursive neural networks for morphology. In CoNLL, pages 104–113. find our card
  13. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  14. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111– 3119. find
  15. Mnih, A. and Kavukcuoglu, K. (2013). Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pages 2265–2273. find
  16. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532– 1543. find
  17. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  18. Schütze, H. (1993). Word space. In Advances in neural information processing systems, pages 895–902. find
  19. Weinberger, K., Dasgupta, A., Langford, J., Smola, A., and Attenberg, J. (2009). Feature hashing for large scale multitask learning. In ICML. find