Elamur Research
☰
home / papers / 2004 / RCV1: A New Benchmark Collection for Text Categorization Research

RCV1: A New Benchmark Collection for Text Categorization Research

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "JMLR — open access, read only" does not allow redistributing the text itself. Original — jmlr.org/papers/volume5/lewis04a/lewis04a.pdf

Summary

Text categorization research has depended on test collections that are small, inconsistently labeled, poorly documented or overused. Overuse lets a whole research community overfit to familiar data. Reuters Corpus Volume I (RCV1) could fill this gap. It holds over 800,000 English newswire stories from August 1996 to August 1997, coded with Topic, Industry and Region categories. As released, however, it is only an archive. It contains coding errors and category lists that disagree with the actual assignments, and it does not explain what the codes mean. The authors drew on Reuters documentation and interviews with staff to reconstruct how stories were coded and quality-checked and to explain the intended meaning of each taxonomy. They also defined a cleaned version, RCV1-v2, as opposed to the original RCV1-v1. They then ran standard supervised learners (SVMs, k-NN and Rocchio-style classifiers) on a fixed training/test split, LYRL2004, to set baselines. SVMs performed best at every category frequency. The corrected labels, hierarchies, vectorized data and per-category results are released as online appendices, giving researchers a large, documented benchmark for reproducible comparison.

Key points

  • RCV1 comes from a Reuters online database. It is meant to cover all English stories by Reuters journalists from August 20, 1996 to August 19, 1997. It is distributed in XML on two CD-ROMs.
  • Topic codes form four hierarchical groups: CCAT, ECAT, GCAT and MCAT. Industry codes form 10 subhierarchies. Region codes have no hierarchy. Coding followed two reconstructed rules: a minimum of one Topic code and one Region code per story, and assignment of the most specific codes together with their ancestors.
  • The paper ships with 18 online appendices. They include valid category lists, the original and expanded Topics hierarchies, RCV1-v2 document IDs and code assignments, tokenized and vectorized data for the LYRL2004 split, and contingency tables of results.
  • SVM.1, which tunes thresholds with SCutFBR.1, matched or beat the costlier leave-one-out SVM.2. Their hyperplanes differed by 19.6 degrees on average, so the effectiveness gap came mainly from threshold choice. Rocchio-style methods trailed the other approaches throughout.
  • The median Topic category has 7,250 test-set occurrences. By January 5, 2004, Reuters had distributed the corpus to 520 groups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

238 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 231 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lewis2004rcv1,
  title = {{RCV1: A New Benchmark Collection for Text Categorization Research}},
  author = {Lewis, David D. and Yang, Yiming and Rose, Tony G. and Li, Fan},
  booktitle = {JMLR 5},
  year = {2004},
  url = {https://www.jmlr.org/papers/v5/lewis04a.html}
}

References

44 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. G. Altman. Practical Statistics for Medical Research. Chapman & Hall/CRC, 1991. find
  2. T. Ault and Y. Yang. kNN, Rocchio and metrics for information filtering at TREC-10. In The Tenth Text REtrieval Conference (TREC 2001), pages 84–93, Gaithersburg, MD 20899-0001, 2002. National Institute of Standards and Technology. http://trec.nist.gov/pubs/trec10/papers/cmucat-correct.pdf. ↗
  3. C. Buckley, G. Salton, and J. Allan. The effect of adding relevance information in a relevance feedback environment. In Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (SIGIR 94), pages 292–300, 1994. find
  4. C. W. Cleverdon. The significance of the Cranfield tests of index languages. In Proceedings of the Fourteenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (SIGIR 91), pages 3–12, 1991. find
  5. Great Britain Office for National Statistics. Indexes to UK Standard Industrial Classification of Economic Activities 1992 UK SIC(92). Office for National Statistics, London, 1997. find
  6. Great Britain Office for National Statistics. UK Standard Industrial Classification of Economic Activities UK SIC(92), December 20, 2002. http://www.statistics.gov.uk/methods quality/sic/contents.asp. ↗
  7. R. Grishman and B. Sundheim. Design of the MUC-6 evaluation. In Sixth Message Understanding Evaluation (MUC-6), pages 1–12. Defense Advanced Research Projects Agency, Morgan Kaufmann, 1995. find
  8. P. J. Hayes and S. P. Weinstein. CONSTRUE/TIS: A System for Content-Based Indexing of a Database of News Stories. In Second Annual Conference on Innovative Applications of Artificial Intelligence, pages 49–64, 1990. find
  9. W. Hersh, C. Buckley, T. J. Leone, and D. Hickman. OHSUMED: an interactive retrieval evaluation and new large text collection for research. In Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (SIGIR 94), pages 192–201, 1994… find
  10. D. J. Ittner, D. D. Lewis, and D. D. Ahn. Text Categorization of Low Quality Images. In Symposium on Document Analysis and Information Retrieval, pages 301–315, Las Vegas, 1995. find
  11. T. Joachims. Text categorization with support vector machines: Learning with many relevant features. In European Conference on Machine Learning (ECML ’98), pages 137–142, Berlin, 1998. find our card
  12. T. Joachims. Transductive inference for text classification using support vector machines. In International Conference on Machine Learning (ICML’99), pages 200–209, San Francisco, CA, 1999. find
  13. T. Joachims. SVM Light: Support Vector Machine, May 13th, 2002. http://svmlight.joachims.org. ↗
  14. D. V. Khmelev and W. J. Teahan. A repetition based measure for verification of text collections and for text categorization. In Proceedings of the Twenty-Sixth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 03), pages 104–110, 2003. find
  15. D. Koller and M. Sahami. Hierarchically classifying documents using very few words. In International Conference on Machine Learning (ICML’97), pages 170–178, Nashville, 1997. find
  16. F. W. Lancaster. Indexing and Abstracting in Theory and Practice. Second edition. University of Illinois, Champaign, IL, 1998. find
  17. D. D. Lewis. Evaluating text categorization. In Proceedings of Speech and Natural Language Workshop, pages 312–318. Defense Advanced Research Projects Agency, Morgan Kaufmann, 1991. find
  18. D. D. Lewis. An evaluation of phrasal and clustered representations on a text categorization task. In Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 92), pages 37–50, 1992. find
  19. D. D. Lewis. Evaluating and optimizing autonomous text classification systems. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 95), pages 246–254, 1995. find
  20. D. D. Lewis. Reuters-21578 text Categorization test collection. Distribution 1.0. README file (version 1.2). Manuscript, September 26, 1997. http://www.daviddlewis.com/resources/testcollections/reuters21578/readme.txt ↗
  21. D. D. Lewis. Applying support vector machines to the TREC-2001 batch filtering and routing tasks. In The Tenth Text REtrieval Conference (TREC 2001), pages 286–292, Gaithersburg, MD 20899-0001, 2002. National Institute of Standards and Technology. http://trec.nist.gov/pubs/trec10/papers/daviddlewis-… ↗
  22. D. D. Lewis, R. E. Schapire, J. P. Callan, and R. Papka. Training algorithms for linear text classifiers. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 96), pages 298–306, 1996. find
  23. D. D. Lewis and R. M. Tong. Text filtering in MUC-3 and MUC-4. In Proceedings of the Fourth Message Understanding Conference (MUC-4), pages 51–66. Defense Advanced Research Projects Agency, Morgan Kaufmann, 1992. find
  24. M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130–137, 1980. find
  25. M. F. Porter. The Porter Stemming Algorithm, 2003. http://www.tartarus.org/˜martin/PorterStemmer. ↗
  26. S. Robertson and I. Soboroff. The TREC 2001 filtering track report. In The Tenth Text REtrieval Conference (TREC 2001), pages 26–37, Gaithersburg, MD 20899-0001, 2002. National Institute of Standards and Technology. http://trec.nist.gov/pubs/trec10/papers/filtering2 track.pdf. ↗
  27. J. J. Rocchio, Jr.. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing, pages 313–323. Prentice-Hall, 1971. find
  28. M. Rogati and Y. Yang. High performing and scalable feature selection for text classification. In Proceedings of the Eleventh International Conference on Information and Knowledge Management, pages 659-661, 2002. find
  29. T. Rose, M. Stevenson, and M. Whitehead. The Reuters Corpus Volume 1 – from Yesterday’s News to Tomorrow’s Language Resources. In Proceedings of the Third International Conference on Language Resources and Evaluation, 2002. http://about.reuters.com/researchandstandards/corpus/LREC camera ready.pdf T… ↗
  30. G. Salton and C. Buckley. Improving retrieval performance by relevance feedback. Journal of American Society for Information Sciences, 41:288–297, 1990. find
  31. G. Salton, editor. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall, 1971. find
  32. R. E. Schapire, Y. Singer, and A. Singhal. Boosting and Rocchio applied to text filtering. In Proceedings of the Twenty-First Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 98), pages 215–223, 1998. find
  33. F. Sebastiani. Machine learning in automated text categorization. ACM Computing Surveys, 34(1):1–47, 2002. find
  34. J. M. Tague. The pragmatics of information retrieval experimentation. In K. Sparck Jones, editor, Information Retrieval Experiment, chapter 5. Butterworths, 1981. find
  35. C. J. van Rijsbergen. Automatic Information Structuring and Retrieval. PhD thesis, King’s College, Cambridge, 1972. find
  36. C. J. van Rijsbergen. Information Retrieval. Butterworths, 1979. find
  37. M. Whitehead. Electronic mail message to ReutersCorpora@yahoogroups.com, November 14, 2002. http://groups.yahoo.com/group/ReutersCorpora/message/106. ↗
  38. A. S. Weigend, E. D. Wiener, and J. O. Pedersen. Exploiting hierarchy in text categorization. Information Retrieval, 1(3):193–216, 1999. find
  39. Y. Yang. An evaluation of statistical approaches to text categorization. Information Retrieval, 1(1/2):67–88, 1999. find
  40. Y. Yang. A study on thresholding strategies for text categorization. In The Twenty-Fourth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 01), pages 137–145, 2001. find
  41. Y. Yang, T. Ault, T. Pierce, and C. W. Lattimer. Improving text categorization methods for event tracking. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 00), pages 65–72, 2000. find
  42. Y. Yang and X. Liu. A re-examination of text categorization methods. In Proceedings of the Twenty-Second Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 99), pages 42–49, 1999. find
  43. Y. Yang and J. O. Pedersen. A comparative study on feature selection in text categorization. In The Fourteenth International Conference on Machine Learning (ICML’97), pages 412–420. Morgan Kaufmann, 1997. find
  44. T. Zhang and F. J. Oles. Text categorization based on regularized linear classification methods. Information Retrieval, 4(1):5–31, 2001. find