Elamur Research⌕
☰
home / papers / 2010 / A Semi-Supervised Approach to Improve Classification of Infrequent Discourse Relations Using Feature Vector Extension

A Semi-Supervised Approach to Improve Classification of Infrequent Discourse Relations Using Feature Vector Extension

Summary

Discourse relation classifiers are usually trained with full supervision. They handle common relations well but perform poorly on rare ones, and annotated data is expensive to produce. Hernault, Bollegala and Ishizuka address this with a semi-supervised technique that draws on cheap unlabeled text. They measure how often features occur together in unlabeled instances, scoring each pair with the χ² statistic. The result is a normalized feature co-occurrence matrix. The matrix is then used to extend feature vectors during both training and testing. As a result, test features never seen in training can still contribute through features related to them. The approach works with any multi-class learner.

Experiments on the RST Discourse Treebank (RSTDT) and Penn Discourse Treebank (PDTB) show large relative gains in accuracy and macro-average F-score when labeled data is scarce. The gains shrink as the labeled set grows. Absolute scores remain too low for standalone parsing. Still, the method is a practical first step toward detecting infrequent relations in domains with little annotation.

Key points

  • Task scale: RSTDT relations become a 41-class problem once nuclearity is included, while PDTB uses 15 level-two relations. Class frequencies range from thousands of instances down to only 2–17.
  • Co-occurrence pairs are kept only when their χ² value exceeds the critical value 3.84 (0.05 confidence level, one degree of freedom). Scores are rescaled to [0, 1].
  • With roughly 1000 labeled instances, accuracy and macro-average F-score improve by up to 50% over a baseline classifier.
  • With 100 labeled and 100,000 unlabeled instances, macro-average F-score rises by 119% on RSTDT and 459% on PDTB. With 10,000 labeled instances, the maximum gains fall to 2.5% and 15% respectively.
  • Improvements grow with the number of test features unseen during training and with the amount of unlabeled data used.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hernault2010semi,
  title = {{A Semi-Supervised Approach to Improve Classification of Infrequent Discourse Relations Using Feature Vector Extension}},
  author = {Hernault, Hugo and Bollegala, Danushka and Ishizuka, Mitsuru},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1039/}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Basili, M. Cammisa, and A. Moschitti. 2006. A semantic kernel to classify texts with very few training examples. Informatica (Slovenia), 30(2):163–172. find
  2. S. Bloehdorn, R. Basili, M. Cammisa, and A. Moschitti. 2006. Semantic kernels for text classification based on topological measures of feature similarity. In Proc. of ICDM’06, pages 808–812. find
  3. L. Carlson, D. Marcu, and M. E. Okurowski. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. Proc. of Second SIGdial Workshop on Discourse and Dialogue-Volume 16, pages 1–10. find our card
  4. N. Cristianini, J. Shawe-Taylor, and H. Lodhi. 2002. Latent semantic kernels. Journal of Intelligent Information Systems, 18:127–152. find
  5. S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391–407. find
  6. D. A. duVerle and H. Prendinger. 2009. A novel discourse parser based on Support Vector Machine classification. In Proc. of ACL’09, pages 665–673. find our card
  7. H. Fang. 2008. A re-examination of query expansion using lexical resources. In Proc. of ACL’08, pages 139– 147. find
  8. C. Fellbaum, editor. 1998. WordNet: An electronic lexical database. MIT Press. find
  9. H. Hernault, D. Bollegala, and M. Ishizuka. 2010. A sequential model for discourse segmentation. In Proc. of CICLing’10, pages 315–326. find
  10. D. Klein and C. D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems, volume 15. MIT Press. find
  11. Z. Lin, M-Y. Kan, and H. T. Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proc. of EMNLP’09, pages 343–351. find our card
  12. E. Loper and S. Bird. 2002. NLTK: The natural language toolkit. In Proc. of ACL’02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics, pages 63–70. find
  13. D. M. Magerman. 1995. Statistical decision-tree models for parsing. Proc. of ACL’95, pages 276–283. find
  14. W. C. Mann and S. A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. find our card
  15. C. D. Manning and H. Schütze. 1999. Foundations of Statistical Natural Language processing. MIT Press. find
  16. D. Marcu and A. Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proc. of ACL’02, pages 368–375. find our card
  17. D. Marcu. 2000. The Theory and Practice of Discourse Parsing and Summarization. MIT Press. find
  18. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  19. N. Okazaki. 2009. Classias: A collection of machine-learning algorithms for classification. http://www.chokkan.org/software/classias/. ↗
  20. B. Pang and L. Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1–135. find
  21. E. Pitler, M. Raghupathy, H. Mehta, A. Nenkova, A. Lee, and A. Joshi. 2008. Easily identifiable discourse relations. In Proc. of COLING’08 (Posters), pages 87–90. find our card
  22. E. Pitler, A. Louis, and A. Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proc. of ACL’09, pages 683–691. find our card
  23. P. Piwek, H. Hernault, H. Prendinger, and M. Ishizuka. 2007. Generating dialogues between virtual agents automatically from text. In Proc. of IVA’07, pages 161– 174. find
  24. R. L. Plackett. 1983. Karl Pearson and the chi-squared test. International Statistical Review / Revue Internationale de Statistique, 51(1):59–72. find
  25. R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008a. The Penn Discourse TreeBank 2.0. In Proc. of LREC’08. find our card
  26. R. Prasad, E. Miltsakaki, N. Dinesh, A. Lee, A. Joshi, L. Robaldo, and B. Webber. 2008b. The Penn Discourse Treebank 2.0 annotation manual. Technical report, University of Pennsylvania Institute for Research in Cognitive Science. find
  27. G. Salton and C. Buckley. 1983. Introduction to Modern Information Retrieval. McGraw-Hill Book Company. find
  28. G. Siolas and F. d’Alché-Buc. 2000. Support Vector Machines based on a semantic kernel for text categorization. In Proc. of IJCNN’00, volume 5, page 5205. find
  29. S. Somasundaran, G. Namata, J. Wiebe, and L. Getoor. 2009. Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In Proc. of EMNLP’09, pages 170–179. find our card
  30. R. Soricut and D. Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. Proc. of NA-ACL’03, 1:149–156. find our card
  31. V. N. Vapnik. 1995. The Nature of Statistical Learning Theory. Springer-Verlag New York, Inc. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hernault-2010-semi-supervised-approach-improve-classification/