Elamur Research⌕
☰
home / papers / 2012 / Improving Word Representations via Global Context and Multiple Word Prototypes

Improving Word Representations via Global Context and Multiple Word Prototypes

Summary

Most word embedding models learn one vector per word from a narrow window of neighbouring words. This ignores two things: words often have several unrelated or related senses, and the surrounding document carries topical cues. Huang et al. address both. They extend the Collobert and Weston ranking-loss language model with a second scoring network whose input is the idf-weighted mean of the embeddings of every word the document contains. The final score adds a local, order-aware component to a global, bag-of-words-like component. The resulting single-prototype embeddings then represent context windows of every word occurrence. These windows are clustered with spherical k-means, occurrences are relabelled by cluster, and a separate vector is trained per sense. Training uses an April 2010 Wikipedia snapshot. The authors also release a new dataset of human similarity ratings for word pairs shown inside sentences, with senses sampled via WordNet. On this dataset the context-aware multi-prototype model scores best (65.7 with AvgSimC), beating C&W vectors and tf-idf baselines. This shows that context-sensitive evaluation and per-sense vectors matter for ambiguous words.

Key points

  • Training data: Wikipedia (April 2010 snapshot, Shaoul and Westbury), about 2M articles and 990M tokens. The vocabulary is the 30,000 most frequent lowercased words. Rare numbers become digit patterns using "DG" or a NUMBER token, and other rare words become UNKNOWN.
  • Setup:
    • 50-dimensional embeddings, 10-word local windows, 100 hidden units, no weight regularization.
    • 10 prototypes per word, built from 5-word windows on each side of an occurrence.
    • Training uses mini-batch L-BFGS with 1000 good/corrupt pairs per batch, which moved embeddings into good positions faster than SGD.
  • New dataset: human judgments on word-pair similarity in sentential context, covering nouns, verbs and adjectives, with senses sampled using WordNet. The model is also evaluated on WordSim-353.
  • Results on the new dataset:
    • Without context, multi-prototype beats single-prototype (62.8 vs. 58.6).
    • AvgSimC with context gives the best score (65.7).
    • Pruned tf-idf (top 200 features) reached 60.5 with AvgSimC in its single-prototype form.
    • Multi-prototype tf-idf did worse (55.4), and movMF clustering reached only 31.5.
  • Resources: the dataset and word vectors were released for download from the authors' Stanford page.

Interesting ideas

  • I1. Adding a document-level score: a cheap global term (a weighted average of document word vectors, concatenated with the target word) is added to a local window score. This injects topical signal without giving up word-order information. The same additive design could enrich embeddings for any language where document metadata or topic matters.
  • I2. Two-stage sense induction: first train ordinary embeddings, then use them to represent and cluster occurrence contexts, then retrain one vector per cluster. This turns any single-vector model into a sense-aware one without labelled sense data, which helps low-resource settings.
  • I3. Evaluating words in context: pairs are rated inside full sentences, and WordNet is used to deliberately sample different senses. Isolated word-pair benchmarks hide polysemy effects, so building a similar resource for another language would make sense-aware models measurable.
  • I4. Context-weighted similarity (AvgSimC): sense-pair similarities are weighted by how likely each sense is given the context. Cluster membership is approximated by inverse distance to the centroid. The method falls back to uniform weights when no context is available, so one metric serves both settings.
  • I5. Digit-pattern normalization: rare numbers are mapped to patterns like "DGDGDGDG" rather than one generic token. This keeps useful shape information, such as "this is a year", while still shrinking the vocabulary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

300 papers of our library cite it (of the 140884 whose bibliographies we read); 31 of them have a page here, 269 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{huang2012improving,
  title = {{Improving Word Representations via Global Context and Multiple Word Prototypes}},
  author = {Huang, Eric and Socher, Richard and Manning, Christopher and Ng, Andrew},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-1092/}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, Jaz K, Thomas Hofmann, Tomaso Poggio, and John Shawe-taylor. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  2. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, ICML ’08, pages 160–167, New York, NY, USA. ACM. find
  3. James Richard Curran. 2004. From distributional to semantic similarity. Technical report. find
  4. Inderjit S. Dhillon and Dharmendra S. Modha. 2001. Concept decompositions for large sparse text data using clustering. Mach. Learn., 42:143–175, January. find
  5. Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. 2011. Multi-view learning of word embeddings via cca. In Advances in Neural Information Processing Systems (NIPS), volume 24. find
  6. Georgiana Dinu and Mirella Lapata. 2010. Measuring distributional similarity in context. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 1162–1172, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  7. Ahmad Emami, Peng Xu, and Frederick Jelinek. 2003. Using a connectionist model in a syntactical based language model. In Acoustics, Speech, and Signal Processing, pages 372–375. find
  8. Katrin Erk and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 897–906, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  9. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. In Proceedings of the 10th international conference on World Wide Web, WWW ’01, pages 406–414, New York, NY, USA. ACM. find
  10. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Proceedings of the 20th international joint conference on Artifical intelligence, IJCAI’07, pages 1606–1611, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  11. Thomas L Griffiths, Kevin R Canini, Adam N Sanborn, and Daniel J Navarro. 2009. Unifying rational models of categorization via the hierarchical dirichlet process. Brain, page 323328. find
  12. David J Hess, Donald J Foss, and Patrick Carroll. 1995. Effects of global and local context on lexical processing during language comprehension. Journal of Experimental Psychology: General, 124(1):62–82. find
  13. Ping Li, Curt Burgess, and Kevin Lund. 2000. The acquisition of word meaning through global lexical co-occurrences. find
  14. D. C. Liu and J. Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Math. Program., 45(3):503–528, December. find
  15. Suresh Manandhar, Ioannis P Klapaftis, Dmitriy Dligach, and Sameer S Pradhan. 2010. Semeval-2010 task 14: Word sense induction & disambiguation. Word Journal Of The International Linguistic Association, (July):63–68. find
  16. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schtze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA. find
  17. George A Miller and Walter G Charles. 1991. Contextual correlates of semantic similarity. Language & Cognitive Processes, 6(1):1–28. find
  18. George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38:39–41. find our card
  19. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In In Proceedings of ACL-08: HLT, pages 236–244. find
  20. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, ICML ’07, pages 641–648, New York, NY, USA. ACM. find
  21. Andriy Mnih and Geoffrey Hinton. 2008. A scalable hierarchical distributed language model. In In NIPS. find
  22. Ht Ng and J Zelle. 1997. Corpus-based approaches to semantic interpretation in natural language processing. AI Magazine, 18(4):45–64. find
  23. Siva Reddy, Ioannis Klapaftis, Diana McCarthy, and Suresh Manandhar. 2011. Dynamic and static prototype vectors for semantic composition. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 705–713, Chiang Mai, Thailand, November. Asian Federation of Natural La… find
  24. Joseph Reisinger and Raymond Mooney. 2010a. A mixture model with sharing for lexical semantics. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 1173–1182, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  25. Joseph Reisinger and Raymond J. Mooney. 2010b. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 109–117, Stroudsburg, PA, USA. Association for… find our card
  26. Yves Rosseel. 2002. Mixture models of categorization. Journal of Mathematical Psychology, 46:178–210. find
  27. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8:627–633, October. find
  28. Hinrich Schütze. 1998. Automatic word sense discrimination. Journal of Computational Linguistics, 24:97– 123. find our card
  29. Holger Schwenk and Jean-luc Gauvain. 2002. Connectionist language modeling for large vocabulary continuous speech recognition. In In International Conference on Acoustics, Speech and Signal Processing, pages 765–768. find
  30. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Comput. Surv., 34:1– 47, March. find
  31. Cyrus Shaoul and Chris Westbury. 2010. The westbury lab wikipedia corpus. find
  32. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 254–263, Stroudsburg, PA, USA. As… find our card
  33. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24. find
  34. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011b. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 26th International Conference on Machine Learning (ICML). find
  35. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011c. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  36. Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. 2003. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th Annual International ACM SIGIR Conference on Search and Development in Information Retrieval, pages 41–47. ACM… find
  37. Stefan Thater, Hagen Fürstenau, and Manfred Pinkal. 2011. Word meaning in context: a simple and effective vector model. In Proceedings of the 5th International Joint Conference on Natural Language Processing, IJCNLP ’11. find
  38. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 384–394, Stroudsburg, PA, USA. Association for Computational… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/