Elamur Research⌕
☰
home / papers / 1999 / Transductive Inference for Text Classification using Support Vector Machines

Transductive Inference for Text Classification using Support Vector Machines

Paper · 1999 · Thorsten Joachims

Summary

Text classifiers are often needed when only a handful of labeled documents exist but a large pool of unlabeled documents is already waiting to be categorized. Examples include relevance feedback, daily netnews filtering, and adding new categories to a company archive. Joachims argues that in these cases learning a general decision rule is unnecessary. Following Vapnik's transductive framework, the learner should instead aim to label that specific test set correctly.

He introduces Transductive Support Vector Machines (TSVMs). They jointly choose labels for the test documents and a hyperplane that separates both labeled and unlabeled data with a wide margin, with slack terms to tolerate errors. The paper explains why margins fit text well: documents are high-dimensional, sparse TF-IDF vectors with strong word co-occurrence patterns. It also gives a training algorithm that scales beyond 10,000 examples.

On three collections, including WebKB and Ohsumed, TSVMs outperform inductive SVMs and Naive Bayes. The gains are largest with tiny training sets and big test sets, and on some tasks TSVMs need only about a twentieth of the labeled data.

Key points

  • Documents are represented as Porter-stemmed bag-of-words vectors with TF-IDF weighting and unit-length normalization. Each category is treated as its own binary problem.
  • The TSVM objective has two parameters, C and C*, which weigh margin size against training errors and against test examples that fall inside the margin.
  • On WebKB, the average P/R-breakeven point was 46.1 for Naive Bayes, 57.2 for SVM and 62.4 for TSVM. For the category course, TSVM jumped to 93.8, compared with 68.7 for SVM.
  • For the category project, TSVM did worse than SVM (18.4 vs. 37.5). With 9 training examples, only one belonged to that class.
  • The benefit grows with test-set size but levels off. On one collection the curve was already flat at about 3,299 test examples.

Interesting ideas

  • I1. Use the unlabeled target pool as a regularizer. When the documents to classify are known in advance, forcing the decision boundary through low-density regions of that pool can replace much of the missing labeled data. This could suit low-resource settings, such as classifying a fixed Ukrainian news archive from a few labeled seeds.
  • I2. Measure gains along two axes. The paper varies training-set size and test-set size separately. This shows both where a semi-supervised method helps most and when adding more unlabeled data stops paying off, a useful evaluation template for any method that exploits unlabeled data.
  • I3. Diagnose per-category failures qualitatively. The weak result on project is explained by a competing high-margin direction, namely the varied project topics. Inspecting what large-margin structure the unlabeled data actually contains is a good habit before trusting margin-based semi-supervised learning.
  • I4. Keep the feature set full. Since most words carry some signal, the paper avoids aggressive feature selection and lets margin-based learning cope with high dimensionality. This is a sensible default for sparse lexical features in morphologically rich languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Machine learning · 51 worksText classification · 50 works

Cite

Cite as Thorsten Joachims. 1999. Transductive Inference for Text Classification using Support Vector Machines. ICML 1999.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{joachims1999transductive,
  title = {{Transductive Inference for Text Classification using Support Vector Machines}},
  author = {Thorsten Joachims},
  booktitle = {ICML 1999},
  year = {1999},
  url = {https://dl.acm.org/doi/10.5555/645528.657646}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bennett, K. (1999). Combining support vector and mathematical programming methods for classi cation. In Scholkopf, B., Burges, C., and Smola, A., editors, Advances in Kernel Methods - Support Vector Learning . MIT-Press. find
  2. Blum, A. and Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Annual Conference on Computational Learning Theory (COLT-98) .
  3. Dumais, S., Platt, J., Heckerman, D., and Sahami, M. (1998). Inductive learning algorithms and representations for text categorization. In Proceedings of ACM-CIKM98 . find
  4. Gammerman, A., Vapnik, V., and Vowk, V. (1998). Learning by transduction. In Conference on Uncertainty in Arti cial Intelligence , pages 148{156. find
  5. Joachims, T. (1997). A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. In Proceedings of International Conference on Machine Learning (ICML) . find
  6. Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In European Conference on Machine Learning (ECML) .
  7. Joachims, T. (1999). Making large-scale svm learning practical. In Scholkopf, B., Burges, C., and Smola, A., editors, Advances in Kernel Methods - Support Vector Learning . MIT-Press. find
  8. McCallum, A. and Nigam, K. (1998). A comparison of event models for naive bayes text classi - cation. In AAAI/ICML Workshop on Learning for Text Classi cation . AAAI Press. find
  9. Mladenic, D. (1998). Feature subset selection in text learning. In European Conference on Machine Learning (ECML) , Springer LNAI. find
  10. Nigam, K., McCallum, A., Thrun, S., and Mitchell, T. (1998). Learning to classify text from labeled and unlabeled documents. In Proceedings of the AAAI-98 . find
  11. Porter, M. (1980). An algorithm for sux stripping. Program (Automated Library and Information Systems) , 14(3):130{137. find
  12. Raghavan, V., Bollmann, P., and Jung, G. (1989). A critical investigation of recall and precision as measures of retrieval system performance. ACM Transactions on Information Systems , 7(3):205{229. find
  13. Salton, G. and Buckley, C. (1988). Term weighting approaches in automatic text retrieval. Information Processing and Management , 24(5):513{523. find
  14. van Rijsbergen, C. (1977). A theoretical basis for the use of co-occurrence data in information retrieval. Journal of Documentation , 33(2):106{119. find
  15. Vapnik, V. (1998). Statistical Learning Theory . Wiley. find
  16. Vapnik, V. and Sterin, A. (1977). On structural risk minimization or overall risk in a problem of pattern recognition. Automation and Remote Control , 10(3):1495{1503. find
  17. Wapnik, W. and Tscherwonenkis, A. (1979). Theorie der Zeichen-erkennung . Akademie Verlag, Berlin. find
  18. Yang, Y. and Pedersen, J. (1997). A comparative study on feature selection in text categorization. In International Conference on Machine Learning (ICML) . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/joachims-1999-transductive-inference-text-classification-svm/