Learning Word Vectors for Sentiment Analysis
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P11-1015.pdf
Summary
Word vectors learned without supervision capture topical similarity well but often miss polarity. In such vectors, "wonderful" and "terrible" can end up close together because they appear in similar contexts. Maas et al. address this with a model that has two parts. The first is an unsupervised probabilistic document model related to LDA. It is log-linear: each word's probability depends on how well its vector aligns with a per-document latent vector θ. The second part is supervised. A logistic regressor must predict document sentiment from the vectors of the words in the document. The two objectives are trained jointly using alternating maximization. The labels come from star ratings that are abundant online. The resulting features were tested with a linear SVM on three tasks: the Pang and Lee polarity corpus, a new IMDB benchmark the authors built, and sentence-level subjectivity detection. In all three, the model beat other vector space models such as LSA and LDA. It worked best when combined with bag-of-words features and given extra unlabeled data. The released IMDB dataset became a standard sentiment benchmark.
Key points
- The model uses a vocabulary of only 5,000 words. Even so, it is competitive with published results on the 2,000-review Pang and Lee (2004) polarity corpus.
- The authors found leakage in that corpus: 1,299 reviews share a movie with another review. About half of each test fold has a same-movie, same-label review in training.
- Large Movie Review Dataset (IMDB): 50,000 strongly polarized reviews, with at most 30 per movie. A rating of ≤4/10 counts as negative and ≥7/10 as positive. It is split 25k/25k with no movie appearing in both training and test.
- On subjectivity detection (10,000 sentences, 10-fold CV), the learned vectors again outperformed the other vector space models.
- The supervised component can also handle continuous, multi-dimensional, or non-sentiment labels.
Cited by
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP
- Generating Natural Language Adversarial Examples
- Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- Pretrained Transformers Improve Out-of-Distribution Robustness
- An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
- Editing Models with Task Arithmetic
- Attention is not Explanation
- Certified Robustness to Adversarial Word Substitutions
- Distributed Representations of Sentences and Documents
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
- TextBugger: Generating Adversarial Text Against Real-world Applications
- Symbolic Chain-of-Thought Distillation: Small Models Can Also ``Think'' Step-by-Step
- Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
- DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
- What Makes Good In-Context Examples for GPT-3?
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
- Bilingual Word Representations with Monolingual Quality in Mind
- Learned in Translation: Contextualized Word Vectors
- Adversarial Training Methods for Semi-Supervised Text Classification
- MTEB: Massive Text Embedding Benchmark
- StereoSet: Measuring stereotypical bias in pretrained language models
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning
- Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
- Using the Output Embedding to Improve Language Models
- Combating Adversarial Misspellings with Robust Word Recognition
- Learning to Generate Reviews and Discovering Sentiment
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
- MPNet: Masked and Permuted Pre-training for Language Understanding
- How to Fine-Tune BERT for Text Classification?
- Long Range Arena: A Benchmark for Efficient Transformers
- UL2: Unifying Language Learning Paradigms
- Steering Language Models With Activation Engineering
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- Attention is not not Explanation
- Unsupervised Data Augmentation for Consistency Training
- Hierarchical Attention Networks for Document Classification
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation
- Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
- Big Bird: Transformers for Longer Sequences
- Word-level Textual Adversarial Attacking as Combinatorial Optimization
- Sentiment Analysis in the Era of Large Language Models: A Reality Check
Cite
@inproceedings{maas2011learning,
title = {{Learning Word Vectors for Sentiment Analysis}},
author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher},
booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
year = {2011},
url = {https://aclanthology.org/P11-1015/}
}References
- C. O. Alm, D. Roth, and R. Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of HLT/EMNLP, pages 579–586. find
- A. Andreevskaia and S. Bergler. 2006. Mining Word-Net for fuzzy sentiment: sentiment tag extraction from WordNet glosses. In Proceedings of the European ACL, pages 209–216. find
- Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. 2003. a neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, August. find
- D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, May. find our card
- J. Boyd-Graber and P. Resnik. 2010. Holistic sentiment analysis across languages: multilingual supervised latent Dirichlet allocation. In Proceedings of EMNLP, pages 45–55. find
- R. Collobert and J. Weston. 2008. A unified architecture for natural language processing. In Proceedings of the ICML, pages 160–167. find
- S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41:391–407, September. find
- R. E. Fan, K. W. Chang, C. J. Hsieh, X. R. Wang, and C. J. Lin. 2008. LIBLINEAR: A library for large linear classification. The Journal of Machine Learning Research, 9:1871–1874, August. find
- J. R. Finkel and C. D. Manning. 2009. Joint parsing and named entity recognition. In Proceedings of NAACL, pages 326–334. find
- A. B. Goldberg and J. Zhu. 2006. Seeing stars when there aren’t many stars: graph-based semi-supervised learning for sentiment categorization. In TextGraphs: HLT/NAACL Workshop on Graph-based Algorithms for Natural Language Processing, pages 45–52. find
- T. Jay. 2000. Why We Curse: A Neuro-Psycho-Social Theory of Speech. John Benjamins, Philadelphia/Amsterdam. find
- D. Kaplan. 1999. What is meaning? Explorations in the theory of Meaning as Use. Brief version — draft 1. Ms., UCLA. find
- A. Kennedy and D. Inkpen. 2006. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22:110–125, May. find
- F. Li, M. Huang, and X. Zhu. 2010. Sentiment analysis with global topics and local dependency. In Proceedings of AAAI, pages 1371–1376. find
- C. Lin and Y. He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceeding of the 18th ACM Conference on Information and Knowledge Management, pages 375–384. find
- J. Martineau and T. Finin. 2009. Delta tfidf: an improved feature space for sentiment analysis. In Proceedings of the 3rd AAAI International Conference on Weblogs and Social Media, pages 258–261. find
- A. Mnih and G. E. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the ICML, pages 641–648. find
- G. Paltoglou and M. Thelwall. 2010. A study of information retrieval weighting schemes for sentiment analysis. In Proceedings of the ACL, pages 1386–1395. find
- B. Pang and L. Lee. 2004. A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271–278. find
- B. Pang and L. Lee. 2005. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL, pages 115– 124. find
- B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP, pages 79–86. find
- C. Potts. 2007. The expressive dimension. Theoretical Linguistics, 33:165–197. find
- B. Snyder and R. Barzilay. 2007. Multiple aspect ranking using the good grief algorithm. In Proceedings of NAACL, pages 300–307. find
- M. Steyvers and T. L. Griffiths. 2006. Probabilistic topic models. In T. Landauer, D McNamara, S. Dennis, and W. Kintsch, editors, Latent Semantic Analysis: A Road to Meaning. find
- J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the ACL, page 384394. find our card
- P. D. Turney and P. Pantel. 2010. From frequency to meaning: vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
- H. Wallach, D. Mimno, and A. McCallum. 2009. Rethinking LDA: why priors matter. In Proceedings of NIPS, pages 1973–1981. find
- C. Whitelaw, N. Garg, and S. Argamon. 2005. Using appraisal groups for sentiment analysis. In Proceedings of CIKM, pages 625–631. find
- T. Wilson, J. Wiebe, and R. Hwa. 2004. Just how mad are you? Finding strong and weak opinion clauses. In Proceedings of AAAI, pages 761–769. find