Elamur Research⌕
☰
home / papers / 2010 / Word Representations: A Simple and General Method for Semi-Supervised Learning

Word Representations: A Simple and General Method for Semi-Supervised Learning

Summary

Supervised NLP systems usually encode words as one-hot vectors, so rare words get poorly estimated parameters and unseen words get none. Complex semi-supervised frameworks force a specific model and training procedure. The authors propose a lighter alternative: induce word representations from unlabeled text once, then add them as extra features to any existing supervised system. They run what they describe as the first controlled comparison of three representations: clusters produced by the Brown algorithm, embeddings following the approach of Collobert and Weston (2008), and the HLBL embeddings of Mnih & Hinton (2009). These were induced on RCV1 and tested on NER and chunking with near-state-of-the-art baselines. Every representation raised accuracy, and combining several gave further gains, larger for NER than for chunking. Brown clusters were strongest for NER because they handle rare words better than embeddings that received few training updates. The authors also give a default scaling rule for embeddings, so these can be used without tuning. They release the features, the code and the full experimental system. The method offers a cheap, model-agnostic route to semi-supervised gains.

Key points

  • Three representations are compared: Brown hierarchical clustering, C&W embeddings and HLBL embeddings. Brown clustering is a class-based bigram model with O(V·K²) cost and includes a 1000-cluster setting. C&W embeddings use a ranking loss that scores real n-grams above corrupted ones.
  • Evaluation covers NER and chunking. For NER, the gains were bigger on out-of-domain test data than in-domain, even though the out-of-domain text was never used for induction.
  • Error analysis by word frequency shows that most NER errors fall on rare words, where Brown clusters err less. For chunking, Brown and C&W make almost identical numbers of errors, mostly on frequent words.
  • With improved training, the authors retract earlier negative findings about C&W embeddings from Turian et al. (2009).
  • Representations, induction code and the NER/chunking system are available at metaoptimize.com/projects/wordreprs. The exact F1 scores are in a section missing from the provided excerpt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cited by

564 papers of our library cite it (of the 140673 whose bibliographies we read); 42 of them have a page here, 522 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{turian2010word,
  title = {{Word Representations: A Simple and General Method for Semi-Supervised Learning}},
  author = {Turian, Joseph and Ratinov, Lev-Arie and Bengio, Yoshua},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1040/}
}

References

52 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ando, R., & Zhang, T. (2005). A high-performance semi-supervised learning method for text chunking. ACL. find
  2. Bengio, Y. (2008). Neural net language models. Scholarpedia, 3, 3881. find
  3. Bengio, Y., Ducharme, R., & Vincent, P. (2001). A neural probabilistic language model. NIPS. find
  4. Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3, 1137–1155. find
  5. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. ICML. find our card
  6. Bengio, Y., & Sénécal, J.-S. (2003). Quick training of probabilistic neural nets by importance sampling. AISTATS. find
  7. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. find our card
  8. Brown, P. F., deSouza, P. V., Mercer, R. L., Pietra, V. J. D., & Lai, J. C. (1992). Class-based n-gram models of natural language. Computational Linguistics, 18, 467–479. find
  9. Candito, M., & Crabbé, B. (2009). Improving generative statistical parsing with semi-supervised word clustering. IWPT (pp. 138–141). find
  10. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML. find
  11. Deschacht, K., & Moens, M.-F. (2009). Semi-supervised semantic role labeling using the Latent Words Language Model. EMNLP (pp. 21–29). find
  12. Dumais, S. T., Furnas, G. W., Landauer, T. K., Deerwester, S., & Harshman, R. (1988). Using latent semantic analysis to improve access to textual information. SIGCHI Conference on Human Factors in Computing Systems (pp. 281–285). ACM. find
  13. Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 781–799. find
  14. Goldberg, Y., Tsarfaty, R., Adler, M., & Elhadad, M. (2009). Enhancing unlexicalized parsing performance using a wide coverage lexicon, fuzzy tag-set mapping, and EM-HMM-based lexical probabilities. EACL. find
  15. Honkela, T. (1997). Self-organizing maps of words for natural language processing applications. Proceedings of the International ICSC Symposium on Soft Computing. find
  16. Honkela, T., Pulkki, V., & Kohonen, T. (1995). Contextual relations of words in grimm tales, analyzed by self-organizing map. ICANN. find
  17. Huang, F., & Yates, A. (2009). Distributional representations for handling sparsity in supervised sequence labeling. ACL. find
  18. Kaski, S. (1998). Dimensionality reduction by random mapping: Fast similarity computation for clustering. IJCNN (pp. 413–418). find
  19. Koo, T., Carreras, X., & Collins, M. (2008). Simple semi-supervised dependency parsing. ACL (pp. 595–603). find our card
  20. Krishnan, V., & Manning, C. D. (2006). An effective two-stage model for exploiting non-local dependencies in named entity recognition. COLING-ACL. find
  21. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 259–284. find
  22. Li, W., & McCallum, A. (2005). Semi-supervised sequence modeling with syntactic topic models. AAAI. find
  23. Liang, P. (2005). Semi-supervised learning for natural language. Master’s thesis, Massachusetts Institute of Technology. find
  24. Lin, D., & Wu, X. (2009). Phrase clustering for discriminative learning. ACL-IJCNLP (pp. 1030–1038). find
  25. Lund, K., & Burgess, C. (1996). Producing highdimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instrumentation, and Computers, 28, 203–208. find
  26. Lund, K., Burgess, C., & Atchley, R. A. (1995). Semantic and associative priming in high-dimensional semantic space. Cognitive Science Proceedings, LEA (pp. 660–665). find
  27. Martin, S., Liermann, J., & Ney, H. (1998). Algorithms for bigram and trigram word clustering. Speech Communication, 24, 19–37. find
  28. Miller, S., Guinness, J., & Zamanian, A. (2004). Name tagging with word clusters and discriminative training. HLT-NAACL (pp. 337–342). find our card
  29. Mnih, A., & Hinton, G. E. (2007). Three new graphical models for statistical language modelling. ICML. find
  30. Mnih, A., & Hinton, G. E. (2009). A scalable hierarchical distributed language model. NIPS (pp. 1081–1088). find
  31. Morin, F., & Bengio, Y. (2005). Hierarchical probabilistic neural network language model. AISTATS. find
  32. Pereira, F., Tishby, N., & Lee, L. (1993). Distributional clustering of english words. ACL (pp. 183–190). find
  33. Ratinov, L., & Roth, D. (2009). Design challenges and misconceptions in named entity recognition. CoNLL. find our card
  34. Ritter, H., & Kohonen, T. (1989). Self-organizing semantic maps. Biological Cybernetics, 241–254. find
  35. Sahlgren, M. (2001). Vector-based semantic analysis: Representing word meanings based on random labels. Proceedings of the Semantic Knowledge Acquisition and Categorisation Workshop, ESSLLI. find
  36. Sahlgren, M. (2005). An introduction to random indexing. Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering (TKE). find
  37. Sahlgren, M. (2006). The word-space model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Doctoral dissertation, Stockholm University. find
  38. Sang, E. T., & Buchholz, S. (2000). Introduction to the CoNLL-2000 shared task: Chunking. CoNLL. find our card
  39. Schwenk, H., & Gauvain, J.-L. (2002). Connectionist language modeling for large vocabulary continuous speech recognition. International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 765–768). Orlando, Florida. find
  40. Sha, F., & Pereira, F. C. N. (2003). Shallow parsing with conditional random fields. HLT-NAACL. find
  41. Spitkovsky, V., Alshawi, H., & Jurafsky, D. (2010). From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. NAACL-HLT. find
  42. Suzuki, J., & Isozaki, H. (2008). Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data. ACL-08: HLT (pp. 665–673). find
  43. Suzuki, J., Isozaki, H., Carreras, X., & Collins, M. (2009). An empirical study of semi-supervised structured conditional models for dependency parsing. EMNLP. find
  44. Turian, J., Ratinov, L., Bengio, Y., & Roth, D. (2009). A preliminary evaluation of word representations for named-entity recognition. NIPS Workshop on Grammar Induction, Representation of Language and Language Learning. find
  45. Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research. find
  46. Ushioda, A. (1996). Hierarchical clustering of words. COLING (pp. 1159–1162). find
  47. Väyrynen, J., & Honkela, T. (2005). Comparison of independent component analysis and singular value decomposition in word context analysis. AKRR’05, International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning. find
  48. Väyrynen, J. J., & Honkela, T. (2004). Word category maps based on emergent features created by ICA. Proceedings of the STeP’2004 Cognition + Cybernetics Symposium (pp. 173–185). Finnish Artificial Intelligence Society. find
  49. Väyrynen, J. J., Honkela, T., & Lindqvist, L. (2007). Towards explicit semantic features using independent component analysis. Proceedings of the Workshop Semantic Content Acquisition and Representation (SCAR). Stockholm, Sweden: Swedish Institute of Computer Science. find
  50. Řehůřek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. LREC. find
  51. Zhang, T., & Johnson, D. (2003). A robust risk minimization based named entity recognition system. CoNLL. find
  52. Zhao, H., Chen, W., Kit, C., & Zhou, G. (2009). Multilingual dependency learning: a huge feature engineering method to semantic dependency parsing. CoNLL (pp. 55–60). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/