Elamur Research⌕
☰
home / papers / 2014 / Improving Lexical Embeddings with Semantic Knowledge

Improving Lexical Embeddings with Semantic Knowledge

Summary

Word vectors trained only on raw text reflect co-occurrence patterns. Those patterns do not always match the kind of semantic similarity a task needs. Yu and Dredze add lexical knowledge to word2vec training to address this. Their Relation Constrained Model (RCM) trains embeddings so that a word can predict the words a resource lists as related to it. Combining RCM with the cbow language-model objective gives a joint model. The authors also test pre-training schemes, including Joint→RCM, where joint-model embeddings initialize a final RCM pass. Training used the NYT 1994–97 portion of Gigaword v5.0 (about 518M tokens) plus lexical PPDB pairs. Evaluation covered three areas: held-out language modeling, synonym retrieval on PPDB and WordNet scored by MRR, and agreement with human paraphrase-quality ratings. The knowledge-enhanced models beat plain cbow almost everywhere. Joint→RCM was strongest on the semantic tasks. The work shows that a cheap extra objective can guide unsupervised embeddings toward a chosen notion of meaning, and the code was released.

Key points

  • Models compared: cbow baseline, RCM (random or cbow-pretrained initialization), the joint objective, and Joint→RCM.
  • Training setup:
    • 200-dimensional vectors.
    • NCE with 15 negative samples drawn from unigram frequency raised to 3/4.
    • Asynchronous multithreaded SGD, with cbow and RCM threads assigned at a 12:1 ratio.
  • Data:
    • PPDB was filtered to the 200k most frequent NYT words and deduplicated.
    • Its highest-accuracy tier, S, was held out as dev/test (1582/1583 pairs).
    • WordNet synonyms gave 1,500 dev and 1,500 test pairs, but WordNet training data was too small to matter, so all RCM training used PPDB.
  • Synonym retrieval (test MRR, 10k candidates):
    • On PPDB, cbow scores 49.31 and Joint→RCM scores 68.20.
    • On WordNet, cbow scores 10.04 and Joint→RCM scores 16.16.
    • More relations helped from XL to XXL, but the noisier XXXL set reduced performance.
  • Other results:
    • NCE perplexity fell from 8.75 (cbow) to 8.20 (Joint-r).
    • The swapped-pair rate against 868 human-rated PPDB pairs dropped from 17.81 to 16.62 with Joint→RCM.

Interesting ideas

  • I1. Balancing objectives through thread allocation. Instead of tuning a weight in a single loss, each objective gets its own share of worker threads, and the main objective's threads decide when training stops. This is a simple way to mix a large unsupervised signal with a small supervised one in any asynchronous training setup.
  • I2. Staged training, from broad to specialized. Embeddings are first learned from text, or jointly, and then refined on the resource alone. This lets words missing from the resource still benefit from corpus statistics. The same idea could adapt general Ukrainian embeddings with a thesaurus or a domain lexicon.
  • I3. Fair comparison when a model cannot do the task directly. To test RCM vectors on language modeling, the authors freeze them as input embeddings and retrain only the remaining parameters with cbow. This avoids penalizing models that were never trained as language models.
  • I4. Retrieval tests split by frequency band. MRR is reported over candidate pools of 10k, 30k and 100k words, which shows how quality drops for rarer vocabulary. This would be useful for morphologically rich languages, where long-tail words make up much of the vocabulary.
  • I5. Using the cleanest data for evaluation, not training. The authors evaluate on the highest-quality tier of an automatically built resource and train on the larger, noisier tiers. They also show that adding more of the noisiest pairs can hurt, which is a reminder to check resource quality rather than simply maximize size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2089

@inproceedings{yu2014improving,
  title = {{Improving Lexical Embeddings with Semantic Knowledge}},
  author = {Yu, Mo and Dredze, Mark},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2089},
  url = {https://doi.org/10.3115/v1/p14-2089}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. Springer. find
  2. Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, et al. 2007. Greedy layer-wise training of deep networks. In Neural Information Processing Systems (NIPS). find
  3. Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In International Conference on Artificial Intelligence and Statistics, pages 127–135. find
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning (ICML). find
  5. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research (JMLR), 11:625–660. find
  6. Christiane Fellbaum. 1999. WordNet. Wiley Online Library. find
  7. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  8. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Association for Computational Linguistics (ACL), pages 873–882. find our card
  9. Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. In Conference on Natural Language Learning (CoNLL). find our card
  10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546. find
  11. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In International Conference on Machine Learning (ICML). find
  12. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426. find
  13. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition. Technical report, Linguistic Data Consortium. find
  14. Deepak Ravichandran, Patrick Pantel, and Eduard Hovy. 2005. Randomized algorithms and nlp: using locality sensitive hash function for high speed noun clustering. In Association for Computational Linguistics (ACL). find
  15. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), pages 1631–1642. find
  16. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Association for Computational Linguistics (ACL). find our card
  17. Benjamin Van Durme and Ashwin Lall. 2010. Online generation of locality sensitive hash signatures. In Association for Computational Linguistics (ACL), pages 231–235. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2014-improving-lexical-embeddings-semantic-knowledge/