Elamur Research
☰
home / papers / 2020 / Implicit Discourse Relation Classification: We Need to Talk about Evaluation

Implicit Discourse Relation Classification: We Need to Talk about Evaluation

What it is about

Results on implicit relation classification in PDTB 2.0 were hard to compare: papers used different label sets (4 top-level senses, 11 or 12 level-2 senses), different test sections (switching sections 21–22 for 23–24 changes the test by 149 items and one label's count by 71, on test sets of about 1,000), different handling of multiply-labelled pairs, and single runs. The authors propose a protocol: section-based cross-validation (12 folds, keeping documents whole so context models remain possible), both label levels, all four label fields counted for multi-label pairs, a fixed level-2 set for PDTB 3.0 (senses with over 100 instances, 14 labels), and means over random restarts. Under it, pre-trained encoders set strong baselines (XLNet-large about 58–61% level-2 accuracy on PDTB 2.0, 64.8% on PDTB 3.0). Single-argument baselines (only Arg1 or only Arg2 shown, like hypothesis-only NLI baselines) measure how much a model can guess without the relation.

Ideas

For Mova

languages/en/36-discourse-relations.md §7 (implicit relations), §11.2 (the learned-rules tension). Code: prag/src/rules.rs, prag/src/fbench.rs (folds, direction, one-argument control). Measures: frame bench cause/next F1 by fold and by direction.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.480.pdf

Summary

Implicit discourse relation classification on the Penn Discourse TreeBank (PDTB) is a popular benchmark, but published numbers are hard to compare. Papers differ in label sets, train/dev/test section splits, how multiply-annotated instances are handled, and whether random-seed variance is reported. Kim et al. catalogue these discrepancies and show that they can shift results by more than the margins used to claim state-of-the-art. They propose a standardized protocol built on cross-validation over whole WSJ sections, which keeps paragraph and document structure intact, and release the preprocessing code. Using this protocol, they fine-tune BERT and XLNet as baselines on PDTB 2.0. They also give the first fine-grained (Level-2) results on PDTB 3.0. XLNet-large significantly beats every previously reported PDTB 2.0 result on the common splits. The analysis finds that the encoders lean heavily on surface lexical cues and are fragile under trivial perturbations. Directional Level-3 senses suffer from skewed distributions, while the distribution of labels within a document carries useful signal. The work provides a fair reference point and motivates better modeling of document-level context.

Key points

  • Proposed protocol: The 25 PDTB sections are split into 12 cross-validation folds (2 dev, 2 test, 21 train) using a sliding window that starts from the Patterson & Kehler split.
    • Results are averaged over 5 random seeds.
    • All four sense fields (SCLASS1A/1B/2A/2B) count as valid labels.
    • PDTB 2.0 L2 uses the standard 11-way label set; PDTB 3.0 L2 keeps the 14 senses with more than 100 instances.
  • Data: PDTB 2.0 has about 43K relations, roughly 16K of them implicit, drawn from over 2K WSJ articles. PDTB 3.0 adds about 13K datapoints and a revised sense hierarchy with directional Level-3 senses.
  • Best model (XLNet-large):
    • PDTB 2.0 L2: 61.29 (Ji split), 58.77 (Lin split), 59.90 (P&K split), and 57.74 cross-validation accuracy.
    • PDTB 3.0 L2: 64.83 cross-validation accuracy.
  • Lexical over-reliance:
    • When Arg2 begins with "to", the true label is PURPOSE 79.9% of the time, yet XLNet-base predicts PURPOSE in 95.8% of such cases.
    • Adding a "-" to the start of spans lowers BERT-large accuracy by about 9 points.
  • Single-span check: Combining predictions from models that see only Arg1 or only Arg2, scored with an oracle, gives an upper bound that no full model reliably exceeds. Code is available at github.com/najoungkim/pdtb3.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.480

@inproceedings{kim2020implicit,
  title = {{Implicit Discourse Relation Classification: We Need to Talk about Evaluation}},
  author = {Kim, Najoung and Feng, Song and Gunasekara, Chulaka and Lastras, Luis},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.480},
  url = {https://doi.org/10.18653/v1/2020.acl-main.480}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571– 583, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  2. Hongxiao Bai, Hai Zhao, and Junhan Zhao. 2019. Memorizing all for implicit discourse relation recognition. arXiv:1908.11317v1. ↗
  3. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2201–2211, Lisbon, Portugal. Association for Computational Linguistics. find
  4. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1726– 1735, Berlin, Germany. Association for Computational Linguistics. find
  5. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 141–151, New Orleans, Louisiana. Association for Computational Linguistics. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Song Feng, Kshitij Fadnis, Q Vera Liao, and Luis A Lastras. 2020. Doc2Dial: a framework for dialogue composition grounded in documents. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20). find
  8. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791, Florence, Italy. Association for Computational Linguistics. find
  9. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 977–986, Baltimore, Maryland. Association for Computational Linguistics. find
  10. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find
  11. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299– 1308, Copenhagen, Denmark. Association for Computational Linguistics. find
  12. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  13. Junyi Jessy Li and Ani Nenkova. 2014. Reducing sparsity improves the recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 199–207, Philadelphia, PA, U.S.A. Association for Computational Linguistics. find
  14. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 343–351, Singapore. Association for Computational Linguistics. find our card
  15. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224–1233, Austin, Texas. Association for Computational Linguistics. find
  16. Annie Louis, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In Proceedings of the SIG-DIAL 2010 Conference, pages 59–62, Tokyo, Japan. Association for Computational Linguistics. find
  17. Linh The Nguyen, Linh Van Ngo, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4201– 4207, Florence, Italy. Association for Computational Linguistics. find
  18. Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 1, Long Papers, pages 281–291, Valencia, Spain. Association for Computational Linguistics. 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510, Florence, Italy. Association for Computational Linguistics. find
  19. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112, Seoul, South Korea. Association for Computational Linguistics. find
  20. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 914–923, Seattle, Washington, USA. Association for Computational Linguistics. find
  21. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–691, Suntec, Singapore. Association for Computational Linguistics. find our card
  22. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In COLING 2008: Companion volume: Posters, pages 87–90, Manchester, UK. COLING 2008 Organizing Committee. find
  23. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM), pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. find
  24. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In LREC 2008. find our card
  25. Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. In LDC2019T05. Philadelphia: Linguistic Data Consortium. find
  26. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006–1017, Vancouver, Canada. Association for Computational Linguistics. find
  27. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087–2097, Brussels, Belgium. Association for Computational Linguistics. find
  28. Wei Shi and Vera Demberg. 2017. Do we need cross validation for discourse relation classification? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 150–156, Valencia, Spain. Association for Computational Linguistics. find
  29. Wei Shi and Vera Demberg. 2019. Learning to explic-itate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden. Association for Computational Linguistics. find
  30. WenTing Wang, Jian Su, and Chew Lim Tan. 2010. Kernel based discourse relation recognition with temporal ordering information. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 710–719, Uppsala, Sweden. Association for Computational Linguistics. find
  31. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16, Beijing, China. Association for Computational Linguistics. find our card
  32. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pre-training for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753– 5763. Curran Associates, Inc. find