Elamur Research
☰
home / papers / 1998 / Text Categorization with Support Vector Machines: Learning with Many Relevant Features

Text Categorization with Support Vector Machines: Learning with Many Relevant Features

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author copy (cs.cornell.edu, LS-8 tech report 23) — read only" does not allow redistributing the text itself. Original — cs.cornell.edu/people/tj/publications/joachims 98a.pdf

Summary

Joachims investigates whether Support Vector Machines suit supervised text categorization, where each category is treated as its own binary decision over bag-of-words vectors with tf-idf weighting. He first argues from the properties of text. Feature spaces are huge (over 10,000 word stems), but almost no features are useless: a naive Bayes model trained only on the lowest-ranked words still beats chance. Document vectors are sparse, and many tasks are linearly separable. Because SVMs control capacity through the margin rather than the number of features, they should cope with this setting without aggressive feature selection. The experiments use Reuters-21578 (ModApte split) and the Ohsumed corpus. SVMs with polynomial and RBF kernels, trained with SVM light, are compared against naive Bayes, Rocchio, k-NN and C4.5, with each baseline given generous tuning. SVMs win on both collections for every parameter choice tested. A VC-dimension-based heuristic also selects good kernel settings automatically. The work established SVMs as a robust, low-tuning standard for text classification.

Key points

  • Data: On Reuters ModApte there are 9603 training and 3299 test documents, 90 categories and 9962 terms. On Ohsumed (1991 abstracts) there are 10,000 training and 10,000 test documents, 23 MeSH "diseases" categories and 15561 terms.
  • Reuters results (microaveraged precision/recall-breakeven):
    • Naive Bayes reaches 72.0, Rocchio 79.9, C4.5 79.4 and k-NN 82.3.
    • With automatically chosen parameters, the polynomial SVM reaches 86.0 and the RBF SVM 86.4.
    • The RBF SVM beats k-NN on 63 of 90 categories, with 19 ties.
  • Ohsumed results:
    • k-NN reaches 59.1, naive Bayes 57.0, Rocchio 56.6 and C4.5 50.0 (C4.5 overfits with more than 500 features).
    • The SVMs reach 65.9 (polynomial) and 66.0 (RBF).
    • The RBF SVM beats k-NN on all 23 categories.
  • Robustness: No overfitting appears even with degree-5 polynomials using all features, so separate feature selection becomes unnecessary.
  • Cost: Training time is similar to C4.5 but slower than naive Bayes, Rocchio and k-NN. Classification is faster than k-NN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

303 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 295 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/bfb0026683

@inproceedings{joachims1998text,
  title = {{Text Categorization with Support Vector Machines: Learning with Many Relevant Features}},
  author = {Joachims, Thorsten},
  booktitle = {ECML 1998},
  year = {1998},
  doi = {10.1007/bfb0026683},
  url = {https://doi.org/10.1007/bfb0026683}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Cortes and V. Vapnik. Support-vector networks. Machine Learning , 20:273{ 297, November 1995. find
  2. T. Joachims. A probabilistic analysis of the rocchio algorithm with t df for text categorization. In International Conference on Machine Learning (ICML) , 1997. find
  3. T. Joachims. Text categorization with support vector machines: Learning with many relevant features. Technical Report 23, Universitat Dortmund, LS VIII, 1997. find
  4. J. Kivinen, M. Warmuth, and P. Auer. The perceptron algorithm vs. winnow: Linear vs. logarithmic mistake bounds when few input variables are relevant. In Conference on Computational Learning Theory , 1995. find
  5. T. Mitchell. Machine Learning . McGraw-Hill, 1997. find
  6. J. R. Quinlan. C4.5: Programs for Machine Learning . Morgan Kaufmann, 1993. find our card
  7. J. Rocchio. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing , pages 313{323. Prentice-Hall Inc., 1971. find
  8. G. Salton and C. Buckley. Term weighting approaches in automatic text retrieval. Information Processing and Management , 24(5):513{523, 1988. find
  9. Vladimir N. Vapnik. The Nature of Statistical Learning Theory . Springer, New York, 1995. find
  10. Y. Yang. An evaluation of statistical approaches to text categorization. Technical Report CMU-CS-97-127, Carnegie Mellon University, April 1997. find
  11. Y. Yang and J. Pedersen. A comparative study on feature selection in text categorization. In International Conference on Machine Learning (ICML) , 1997. This article was processed using the L A TEX macro package with LLNCS style find