Elamur Research⌕
☰
home / papers / 2013 / Linguistic Regularities in Continuous Space Word Representations

Linguistic Regularities in Continuous Space Word Representations

Summary

The paper asks whether word vectors learned as a side effect of neural language modeling encode linguistic structure in a simple, usable form. The authors take the input-layer weights of a recurrent neural network language model (RNNLM) trained on 320M words of Broadcast News and test them against two analogy benchmarks. The first is a new syntactic set of 8000 questions built from Penn Treebank-tagged newspaper text. The second is SemEval-2012 Task 2, which covers semantic relation similarity. To answer an analogy a:b :: c:?, they add the difference b−a to c and pick the vocabulary word whose vector has the highest cosine similarity to the result.

Relations turn out to behave like roughly constant directions in the embedding space. The largest RNN model answers close to 40% of the syntactic questions, far ahead of LSA. On the semantic task it beats the strongest earlier system, even though the vectors were never tuned for that task. This made vector arithmetic a standard way to probe and evaluate embeddings.

Key points

  • The vectors come from an RNNLM, where each word's representation is a column of the input matrix U. The models were built with Mikolov's RNN toolkit, with an 82k vocabulary and sizes of 80, 320, 640 and a combined 1600 dimensions.
  • The syntactic test set was built from the 100 most frequent JJR, NNS, NN POS and VB words found in 267M tagged words. It covers adjective degrees, noun number, possessives and verb forms, totals 8000 questions, and was released publicly.
  • Syntactic accuracy reaches 39.6% for RNN-1600, with 62.2% on verbs. The best LSA model trained on the same data scores 16.5%.
  • The authors also compared against Turian's embeddings on a shared 36k vocabulary with 6632 questions. RNN-80 scores 19.0%, HLBL-100 scores 18.7% despite training on only 37M words, and CW vectors stay at or below 5.0%.
  • On the 69 test relations of SemEval-2012 Task 2, RNN-1600 reaches Spearman's ρ of 0.275 and MaxDiff accuracy of 0.418. The previous best system, UTD-NB, scores 0.230 and 0.395, and RNN-320 and RNN-640 also beat it.

Interesting ideas

  • I1. Relations as offsets: the method treats a linguistic relation as a fixed displacement vector, so analogy questions need nothing more than addition, subtraction and cosine search. This makes a cheap, training-free probe for any embedding space, for example to check whether Ukrainian vectors capture case or aspect pairs.
  • I2. Benchmarks generated from a tagged corpus: analogy questions are produced automatically by pairing frequent words of the same POS category and including both directions of each pair. The same recipe could quickly yield morphology test sets for languages with an existing tagger or treebank.
  • I3. One scoring rule for two task types: the same cosine expression either retrieves a missing word or rates a given candidate pair, and graded relation similarity is computed by averaging the score over the prototype pairs. This lets one embedding serve both open-ended and ranking evaluations without task-specific training.
  • I4. Fair comparison on a shared vocabulary: embeddings from different sources are compared only after restricting the test set to the words all of them contain. This simple control is worth copying whenever resources with mismatched vocabularies are benchmarked against each other.
  • I5. Concatenating several models: joining vectors from different models into one 1600-dimensional composite gave the best scores. Ensembling at the representation level can be an easy gain before trying any new architecture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

728 papers of our library cite it (of the 140884 whose bibliographies we read); 59 of them have a page here, 669 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mikolov2013linguistic,
  title = {{Linguistic Regularities in Continuous Space Word Representations}},
  author = {Mikolov, Tomas and Yih, Wen-tau and Zweig, Geoffrey},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1090/}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, R. Ducharme, Vincent, P., and C. Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Reseach, 3(6). find
  2. Y. Bengio, H. Schwenk, J.S. Senécal, F. Morin, and J.L. Gauvain. 2006. Neural probabilistic language models. Innovations in Machine Learning, pages 137–186. find
  3. A. Bordes, X. Glorot, J. Weston, and Y. Bengio. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of 15th International Conference on Artificial Intelligence and Statistics. find
  4. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  5. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(96). find
  6. J.L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2):195–225. find
  7. G.E. Hinton and R.R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507. find
  8. G. Hinton and R. Salakhutdinov. 2010. Discovering binary codes for documents by learning deep generative models. Topics in Cognitive Science, 3(1):74–91. find
  9. G.E. Hinton. 1986. Learning distributed representations of concepts. In Proceedings of the eighth annual conference of the cognitive science society, pages 1–12. Amherst, MA. find
  10. David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics (SemEval 2012), pages 356–364. Association for Computational Linguistics. find
  11. Hai-Son Le, I. Oparin, A. Allauzen, J.-L. Gauvain, and F. Yvon. 2011. Structured output layer neural network language model. In Proceedings of ICASSP 2011. find
  12. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2):313–330. find our card
  13. Tomas Mikolov, Martin Karafiat, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of Interspeech 2010. find
  14. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget, and Jan Cernocky. 2011a. Strategies for Training Large Scale Neural Network Language Models. In Proceedings of ASRU 2011. find
  15. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2011b. Extensions of recurrent neural network based language model. In Proceedings of ICASSP 2011. find
  16. Tomas Mikolov. 2012. RNN toolkit. find
  17. A. Mnih and G.E. Hinton. 2009. A scalable hierarchical distributed language model. Advances in neural information processing systems, 21:1081–1088. find
  18. F. Morin and Y. Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252. find
  19. J.B. Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46(1):77–105. find
  20. Bryan Rink and Sanda Harabagiu. 2012. UTD: Determining relational similarity using lexical patterns. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics (SemEval 2012), pages 413–418. Association for Computational Linguistics. find
  21. Holger Schwenk. 2007. Continuous space language models. Computer Speech and Language, 21(3):492 – 518. find
  22. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of Association for Computational Linguistics (ACL 2010). find our card
  23. P.D. Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533–585. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/