Elamur Research⌕
☰
home / papers / 2017 / On the Need of Cross Validation for Discourse Relation Classification

On the Need of Cross Validation for Discourse Relation Classification

Summary

Implicit discourse relation classification on the Penn Discourse Treebank (PDTB) is usually evaluated by training on sections 2–21 and testing on section 23. That test section holds only 761 implicit relations. Shi and Demberg argue that this is too few to tell whether a new feature genuinely helps, especially for the 11-way second-level sense distinction. To show this, they built an LSTM classifier modeled on Rutherford et al. (2017). It encodes both arguments with Word2Vec embeddings and then concatenates handcrafted surface features before a softmax layer. They tested five popular feature types under two protocols: the standard split, and 10-fold cross validation over sections 0–24 with class-balanced folds. On section 23, every feature appeared to improve accuracy. Under cross validation, most features helped in only a minority of folds or had no net effect. The authors conclude that the field risks adopting features that merely fit one small test set. They recommend cross validation, combined with properly reported significance testing, as standard practice.

Key points

  • With 761 test relations in section 23, one accuracy point equals only about 7–8 additional correct instances. Class proportions also differ between the standard train and test sets; for example, temporal relations are under-represented in the test set.
  • The cross-validation setup uses 10 balanced folds over PDTB sections 0–24, averaging about 14,045 training and 1,565 test instances per fold. It covers 11 second-level relations after rare classes were dropped.
  • The model is an LSTM with surface features, implemented in Keras on Theano. It reaches 37.68% on section 23 without extra features, which is competitive with Lin et al. (2009) at 40.20% and Ji & Eisenstein (2015) at 40.66%.
  • Inquirer Tags added about 2.8 points on the standard split. Under cross validation they improved only 2 of 10 folds, and mean accuracy fell slightly from 34.44% to 33.58%. Verb features also looked helpful on section 23 but split 5/5 across folds.
  • Accuracy on the standard test section was generally 3–7 points higher than on the rest of the corpus. The authors favor cross-validation-based significance testing over bootstrapping on the single standard test set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shi2017need,
  title = {{On the Need of Cross Validation for Discourse Relation Classification}},
  author = {Shi, Wei and Demberg, Vera},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.22028/d291-30985},
  url = {https://doi.org/10.22028/d291-30985}
}

References

34 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, volume3:329– 344. find our card
  2. Yoshua Bengio and Yves Grandvalet. 2005. Bias in estimating the variance of k-fold cross-validation. In Statistical modeling and analysis for complex data problems, pages 75–95. Springer. find
  3. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In Proceedings of NAACL-HLT 2016, pages 332–342, San Diego, California. Association for Computational Linguistics. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. journal of machine learning research, volume3:1137–1155. find
  5. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in nlp. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea.… find
  6. Ji-Hyun Kim. 2009. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics & Data Analysis, volume53(11):3735–3745. find
  7. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 343–351, Singapore. Association for Computational Linguistics. find our card
  8. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 69–73, Sofia, Bulgaria. Association for Computational Linguistics. find our card
  9. Thomas Meyer, Najeh Hajlaoui, and Andrei Popescu-Belis. 2015. Disambiguating discourse connectives for statistical machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(7):1184–1197. find
  10. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Empirical Methods in Natural Language Processing (EMNLP 2015), pages 2201– 2211, Lisbonne, Portugal. Association for Computational Linguistics. find our card
  11. Claudiu Mihăilă and Sophia Ananiadou. 2014. Semi-supervised learning of causal relations in biomedical scientific discourse. Biomedical engineering online, 13(2):1. find
  12. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1726–1735, Berlin, Germany. Ass… find our card
  13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  14. Hugo Hernault, Danushka Bollegala, and Mitsuru Ishizuka. 2010. A semi-supervised approach to improve classification of infrequent discourse relations using feature vector extension. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 399–409, MIT, Massach… find our card
  15. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 108–112, Seoul, South Korea. Association for Computational Ling… find our card
  16. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K. Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING2008), pages 85– 88, Manchester, UK. find our card
  17. Christopher Hidey and Kathleen McKeown. 2016. Identifying causal relation using parallel wikipedia articles. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1424–1433, Berlin,Germany. Association for Computational Linguistics. find
  18. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume… find our card
  19. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 977–986, Baltimore, Maryland, USA. Association for Computational Linguistic… find
  20. Conll 2016 shared task on multilingual shallow discourse parsing. pages 1–19, Berlin, Germany. Association for Computational Linguistics. find our card
  21. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L. Webber. 2008. The penn discourse treebank 2.0. In In Proceedings of the 6th International Conference on Language Resources and Evaluation(LREC), Marrakech, Morocco. European Language Resources A… find our card
  22. Chris Quirk and Hoifung Poon. 2016. Distant supervision for relation extraction beyond the sentence boundary. arXiv preprint arXiv:1609.04873. find
  23. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1834–1839, Doha, Qatar. Association for Computational L… find our card
  24. Ines Rehbein, Merel Scholman, and Vera Demberg. 2016. Annotating discourse relations in spoken language: A comparison of the pdtb and ccr frameworks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 23–28, Portoro, Slovenia. European Langua… find our card
  25. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Lisbon, Portugal. Association… find our card
  26. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In European Chapter of the Association for Computational Linguistics.(EACL), pages 645– 654, Gothenburg, Sweden. Association for Computational Lingui… find our card
  27. Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Neural discourse relation recognition with semantic memory. arXiv preprint arXiv:1603.03873. — [arXiv:1603.03873](https://arxiv.org/abs/1603.03873) find
  28. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1507–1514, Beijing, China. Association for Compu… find our card
  29. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the NAACL-HLT, pages 799–808, Denver, Colorado. Association for Computational Linguistics. find our card
  30. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In European Chapter of the Association for Computational Linguistics.(EACL), Valencia, Spain. Association for Computational Linguistics. find our card
  31. Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 149–156, Edmonton. Associat… find our card
  32. Xun Wang, Sujian Li, Jiwei Li, and Wenjie Li. 2012. Implicit discourse relation recognition by selecting typical training examples. In Proceeding of COL-ING 2012, pages 2757–2772, Mumbai. find our card
  33. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad Christopher Bryant, and Attapol T. Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of Conference on Computational Natural Language Learning: Shared Task, pages 1–16, Beijing, China. Association for Comp… find our card
  34. Nianwen Xue, Hwee Tou Ng, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2017-need-cross-validation-discourse-relation/