Elamur Research⌕
☰
home / papers / 2016 / Learning to Understand Phrases by Embedding the Dictionary

Learning to Understand Phrases by Embedding the Dictionary

Summary

Learning vector representations for whole phrases is hard, mainly because there is no obvious gold-standard target to train or evaluate them against. Hill, Cho, Korhonen and Bengio address this with dictionaries. A definition should carry roughly the meaning of its headword, so pre-trained Word2Vec vectors of headwords can supervise phrase encoders. They train two families of neural language models (NLMs) to project a definition close to the embedding of the word it defines. One is an LSTM-based RNN and the other a plain additive bag-of-words (BOW) model. Training uses cosine distance or a margin-based rank loss. Input word vectors are either learned from scratch or fixed to Word2Vec. The data come from a few freely available lexical resources, plus Wikipedia opening sentences for one task. The models are applied as a reverse dictionary (description to word) and as a solver for general-knowledge, non-cryptic crossword clues. They use far less dictionary data and no hand-crafted features, yet match or beat commercial tools. The simple BOW model proves surprisingly competitive. The authors propose definition-to-word mapping as both a training signal and a benchmark for compositional semantics.

Key points

  • Architectures and training: an LSTM RNN (with a final projection into the target space) and a linear BOW encoder. Both map definitions onto Word2Vec embeddings learned from billions of words. The rank loss uses a margin of 0.1. Training took 6–10 hours, and no hyper-parameter search was performed.
  • Reverse dictionary: results are comparable to or better than commercial systems such as OneLook.com, and the candidate lists are smoother. Bilingual embeddings allow a cross-lingual variant.
  • Crossword QA: trained on definitions plus Wikipedia first sentences, the NLMs outperform the commercial solvers One Across and Crossword Maestro on clues longer than four words.
  • Generalisation example: the BOW model returns Schoenberg for an "atonality" clue, even though its single training definition shares only the word "composer" with the query.
  • Release: code, training and evaluation sets, and a demo are publicly released. The paper appeared in TACL vol. 4 (2016), pp. 17–30.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00080

@article{hill2016learning,
  title = {{Learning to Understand Phrases by Embedding the Dictionary}},
  author = {Hill, Felix and Cho, Kyunghyun and Korhonen, Anna and Bengio, Yoshua},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00080},
  url = {https://doi.org/10.1162/tacl_a_00080}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceeding of ICLR. find our card
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  3. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the Association for Computational Linguistics. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio.… find
  4. Slaven Bilac, Timothy Baldwin, and Hozumi Tanaka. 2003. Improving dictionary accessibility by maximizing use of available knowledge. Traitement Automatique des Langues, 44(2):199–224. find
  5. Slaven Bilac, Wataru Watanabe, Taiichi Hashimoto, Takenobu Tokunaga, and Hozumi Tanaka. 2004. Dictionary search based on the target word description. In Proceedings of NLP 2014. find
  6. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. Proceedings of EMNLP. find
  7. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075. find
  8. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C. Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems, pages 1853–1861. Kyunghyun Cho, Bart Van Merriënboe… find
  9. Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2014. Retrofitting word vectors to semantic lexicons. Proceedings of the North American Chapter of the Association for Computational Linguistics. find
  10. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building Watson: An overview of the DeepQA project. In AI magazine, volume 31(3), pages 59–79. find
  11. Matthew L. Ginsberg. 2011. Dr. FILL: Crosswords and an implemented solver for singly weighted CSPs. In Journal of Artificial Intelligence Research, pages 851– 886. find
  12. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2014. BilBOWA: Fast bilingual distributed representations without word alignments. In Proceedings of NIPS Deep Learning Workshop. find
  13. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  14. Karl Moritz Hermann and Phil Blunsom. 2013. Multilingual distributed representations without word alignment. In Proceedings of ICLR. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. find
  15. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the Association for Computational Linguistics. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal… find our card
  16. Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2015. Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics. to appear. find our card
  17. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. Proceedings of COLING. find our card
  18. Geoffrey Leech, Roger Garside, and Michael Bryant. 1994. CLAWS4: The tagging of the British National Corpus. In Proceedings of COLING. find
  19. Michael L. Littman, Greg A. Keim, and Noam Shazeer. 2002. A probabilistic approach to solving crossword puzzles. Artificial Intelligence, 134(1):23–55. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceed… find
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. find
  21. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. Diego Mollá and José Luis Vicedo. 2007. Question answering in restricted domains: An overview. Computational Linguistics, 33(1):41–61. find
  22. Ryan Shaw, Anindya Datta, Debra VanderMeer, and Kaushik Dutta. 2013. Building a scalable database-driven reverse dictionary. Knowledge and Data Engineering, IEEE Transactions on, 25(3):528–540. find
  23. Ivan Vulic, Wim De Smet, and Marie-Francine Moens. 2011. Identifying word translations from comparable corpora using latent topic models. In Proceedings of the Association for Computational Linguistics. find
  24. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. In arXiv preprint arXiv:1502.05698. find our card
  25. Matthew D. Zeiler. 2012. Adadelta: An adaptive learning rate method. In arXiv preprint arXiv:1212.5701. find our card
  26. Michael Zock and Slaven Bilac. 2004. Word lookup on the basis of associations: From an idea to a roadmap. In Proceedings of the ACL Workshop on Enhancing and Using Electronic Dictionaries. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hill-2016-learning-understand-phrases-embedding-dictionary/