Elamur Research⌕
☰
home / papers / 2012 / Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Summary

Naive Bayes and linear SVMs are common baselines in text classification, but their accuracy depends strongly on the model variant, the feature set and the dataset. Many published comparisons therefore rely on weak versions of them. Wang and Manning test these choices systematically on nine binary tasks. Six are sentiment or subjectivity tasks with short snippets or full reviews (RT-s, MPQA, CR, Subj., RT-2k, IMDB), and three are 20-newsgroups topic pairs. They compare binarized multinomial NB, an L2-regularized L2-loss SVM (LIBLINEAR) and a new hybrid called NBSVM, each with unigrams or unigrams plus bigrams. NBSVM is an SVM trained on features scaled by NB log-count ratios, with its weights pulled toward their mean magnitude. The main findings are:

  • Bigrams reliably help sentiment classification.
  • MNB beats SVM on short snippets, while SVM wins on long reviews.
  • NBSVM is strong everywhere and often matches or exceeds more complex published systems, such as recursive autoencoders and tree-based CRFs.

This matters because it gives a cheap, reproducible baseline that sophisticated models must clearly beat.

Key points

  • Results: NBSVM-bi scores 91.22% on IMDB, 89.45% on RT-2k, 79.4% on RT-s and 81.8% on CR. These match or exceed earlier systems that use lexicons, reversal rules, parse trees or Wikipedia pretraining.
  • Snippets versus long reviews: On snippets, MNB outperforms the linear SVM on every dataset except MPQA, even with roughly 9k training examples. On the long-review sets RT-2k and IMDB, the SVM is clearly stronger than MNB.
  • Bigrams: Adding bigrams always improved the sentiment results. On the newsgroup topic tasks the effect was mixed and small, and adding trigrams slightly hurt.
  • Settings: Every reported run shares one set of hyperparameters: α = 1, with NBSVM using C = 1 and β = 0.25 and the SVM using C = 0.1. No stopword lists, lexicons or external resources are used.
  • NB variants: Multivariate Bernoulli NB is less stable than MNB and can be up to 10% worse. Binarizing counts gives about 1% over raw counts on long documents and makes little difference on snippets.

Interesting ideas

  • I1. Generative weights as discriminative features: Multiplying binary features by NB log-count ratios before training a discriminative model is a cheap hybrid. It could boost any linear classifier, for example on low-resource Ukrainian text data.
  • I2. Interpolation as regularization: Mixing the learned SVM weights with a constant equal to their average magnitude means the model follows NB unless the SVM is confident. Values of β between ¼ and ½ were robust, so a similar shrinkage trick could stabilize other models.
  • I3. Strong baselines first: Tuning simple bag-of-features baselines properly can overturn claims that structure-sensitive models are required. New methods for sentiment or classification should be benchmarked against such tuned baselines.
  • I4. Report significance thresholds per dataset: Listing the accuracy gap needed for significance at p < 0.05 for each dataset makes it obvious which improvements are real. This is a good habit for any benchmark table.
  • I5. Choose features by task type: Bigrams capture modified words such as negated verbs and adjectives, which matters for sentiment but much less for topic labelling. Feature choices should therefore follow the task rather than habit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

172 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 154 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wang2012baselines,
  title = {{Baselines and Bigrams: Simple, Good Sentiment and Topic Classification}},
  author = {Wang, Sida and Manning, Christopher},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-2018/}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. find
  2. George E. Dahl, Ryan P. Adams, and Hugo Larochelle. 2012. Training restricted boltzmann machines on word observations. arXiv:1202.5695v1 [cs.LG]. — [arXiv:1202.5695](https://arxiv.org/abs/1202.5695) find
  3. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874, June. find
  4. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings ACM SIGKDD, pages 168–177. find
  5. Alistair Kennedy and Diana Inkpen. 2006. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22. find
  6. Simon Lacoste-Julien, Fei Sha, and Michael I. Jordan. 2008. DiscLDA: Discriminative learning for dimensionality reduction and classification. In Proceedings of NIPS, pages 897–904. find
  7. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL. find our card
  8. Justin Martineau and Tim Finin. 2009. Delta tfidf: An improved feature space for sentiment analysis. In Proceedings of ICWSM. find
  9. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for naive bayes text classification. In AAAI-98 Workshop, pages 41–48. find
  10. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive bayes - which naive bayes? In Proceedings of CEAS. find
  11. Karo Moilanen and Stephen Pulman. 2007. Sentiment composition. In Proceedings of RANLP, pages 378– 382, September 27-29. find
  12. Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In Proceedings of ACL:HLT. find
  13. Andrew Y Ng and Michael I Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Proceedings of NIPS, volume 2, pages 841–848. find
  14. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL. find our card
  15. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL. find our card
  16. Jason D. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger. 2003. Tackling the poor assumptions of naive bayes text classifiers. In Proceedings of ICML, pages 616–623. find
  17. Greg Schohn and David Cohn. 2000. Less is more: Active learning with support vector machines. In Proceedings of ICML, pages 839–846. find
  18. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of EMNLP. find our card
  19. Casey Whitelaw, Navendu Garg, and Shlomo Argamon. 2005. Using appraisal taxonomies for sentiment analysis. In Proceedings of CIKM-05. find
  20. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165–210. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/