Elamur Research⌕
☰
home / papers / 1998 / A Comparison of Event Models for Naive Bayes Text Classification

A Comparison of Event Models for Naive Bayes Text Classification

Paper · 1998 · Andrew McCallum and Kamal Nigam

Summary

Two different probabilistic classifiers were both being called "naive Bayes" in text categorization research. One is the multi-variate Bernoulli model, which treats a document as a binary vector recording which vocabulary words appear. The other is the multinomial model, a unigram language model that counts how often each word occurs. McCallum and Nigam give a shared mixture-model derivation of both, showing where their assumptions and parameter estimates differ. They then test both on five corpora drawn from Web pages, UseNet postings and Reuters newswire, varying vocabulary size through mutual-information feature selection. The Bernoulli variant sometimes wins when the vocabulary is very small. The multinomial generally pulls ahead as vocabulary grows and almost always wins when each model is given its best vocabulary size. Averaged over datasets, it reduces error by 27%. This matters because many real categories cover diverse topics and need large vocabularies, so the paper gives practitioners a clear reason to prefer word counts over presence/absence features.

Key points

  • The two models:
    • The Bernoulli model multiplies probabilities over the whole vocabulary, including explicit terms for words missing from a document.
    • The multinomial model uses only the words that appear, weighted by their counts, and includes document length as the number of draws.
  • Smoothing and priors: Both models smooth word estimates with a Laplacean (add-one) prior. Class priors are maximum-likelihood estimates.
  • Feature selection: Words are ranked by average mutual information with the class variable to shrink the vocabulary.
  • Main result: The multinomial cuts error by 27% on average across the five corpora, and by more than 50% in some cases. Data includes Reuters, Yahoo! and the Market Guide Industry Sector hierarchy.
  • Reuters behaviour: Reuters tasks often peak with small vocabularies. The authors' scores there exceed those of Joachims (1998), who mixed event models between feature selection and classification.

Interesting ideas

  • I1. Compare models at their own best settings. Each model is evaluated at its own optimal vocabulary size, not only at a shared fixed size. This gives a fairer head-to-head comparison for any method that depends on a tunable feature budget.
  • I2. Token normalization combined with counts. All digit strings are collapsed into one token. Its presence is useless, since every article is dated, but its frequency separates earnings reports from other news. This trick could help with Ukrainian trade or financial texts.
  • I3. Keep non-text features out of the word vocabulary. Metadata features such as a message's recipient count should not compete with words for the same probability mass. They can be modelled as separate Bernoulli variables next to the word multinomial, optionally with a weight, which is a clean way to fuse metadata with text.
  • I4. Absent words still carry evidence. In the multinomial, a missing high-probability class word lowers that class's likelihood implicitly. This is a useful argument against assuming explicit absence features are always needed.
  • I5. Model document length explicitly. Making length part of the generative story, and possibly class-dependent, matters for corpora with very uneven document lengths. Under the Bernoulli model, long documents tend to switch on many words regardless of their class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text classification · 50 works

Cite

Cite as Andrew McCallum and Kamal Nigam. 1998. A Comparison of Event Models for Naive Bayes Text Classification. AAAI-98 Workshop on Learning for Text Categorization.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mccallum1998comparison,
  title = {{A Comparison of Event Models for Naive Bayes Text Classification}},
  author = {Andrew McCallum and Kamal Nigam},
  booktitle = {AAAI-98 Workshop on Learning for Text Categorization},
  year = {1998},
  url = {https://aaai.org/papers/041-ws98-05-007/}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley, 1991. M. Craven, D. DiPasquo, D. Freitag, A. McCallum, T. Mitchell, K. Nigam, and S. Slattery. Learning to extract symbolic knowledge from the World Wide Web. In AAAI-98, 1998. P. Domingos and M. Pazzani. On the optimalit… find
  2. Jerome H. Friedman. On bias, variance, 0/1 - loss, and the curse-of-dimensionality. Data Mining and Knowledge Discovery, 1:55–77, 1997. Louise Guthrie and Elbert Walker. Document classification by machine: Theory and practice. In Proceedings of COLING-94, 1994. find
  3. Thorsten Joachims. A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. In ICML-97, 1997. find
  4. Thorsten Joachims. Text categorization with Support Vector Machines: Learning with many relevant features. In ECML-98, 1998.
  5. T. Kalt and W. B. Croft. A new probabilistic model of text classification and retrieval. Technical Report IR-78, University of Massachusetts Center for Intelligent Information Retrieval, 1996. http://ciir.cs.umass.edu/publications/index.shtml. ↗
  6. Daphne Koller and Mehran Sahami. Hierarchically classifying documents using very few words. In Proceedings of the Fourteenth International Conference on Machine Learning, 1997. find
  7. Pat Langley, Wayne Iba, and Kevin Thompson. An analysis of Bayesian classifiers. In AAAI-92, 1992. find
  8. Leah S. Larkey and W. Bruce Croft. Combining classifiers in text categorization. In SIGIR-96, 1996. find
  9. D. Lewis and W. Gale. A sequential algorithm for training text classifiers. In SIGIR-94, 1994. find
  10. David D. Lewis. An evaluation of phrasal and clustered representations on a text categorization task. In SIGIR-92, 1992. find
  11. David Lewis. Naive (bayes) at forty: The independence asssumption in information retrieval. In ECML’98: Tenth European Conference On Machine Learning, 1998. find
  12. Hang Li and Kenji Yamanishi. Document classification using a finite mixture model. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, 1997. find
  13. Ray Liere and Prasad Tadepalli. Active learning with committees for text categorization. In AAAI-97, 1997. find
  14. Andrew McCallum, Ronald Rosenfeld, Tom Mitchell, and Andrew Ng. Improving text clasification by shrinkage in a hierarchy of classes. In ICML-98, 1998. find
  15. Tom M. Mitchell. Machine Learning. WCB/McGraw-Hill, 1997. find
  16. Kamal Nigam, Andrew McCallum, Sebastian Thrun, and Tom Mitchell. Learning to classify text from labeled and unlabeled documents. In AAAI-98, 1998. find
  17. S. E. Robertson and K. Sparck-Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129–146, 1976. find
  18. Mehran Sahami, Susan Dumais, David Heckerman, and Eric Horvitz. A bayesian approach to filtering junk e-mail. In AAAI-98 Workshop on Learning for Text Categorization, 1998. find
  19. Mehran Sahami. Learning limited dependence Bayesian classifiers. In KDD-96: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 335–338. AAAI Press, 1996. find
  20. V. Vapnik. Estimations of dependences based on statistical data. Springer Publisher, 1982. find
  21. Yiming Yang and Jan Pederson. Feature selection in statistical learning of text categorization. In ICML-97, 1997. acq Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 find
  22. Vocabulary Size crude Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary Size grain Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary Size trade Multinomial Multi-variate Bernoulli Precision/Recall Breakeven Point 1000 Vocabulary S… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mccallum-1998-comparison-event-models-naive-bayes/