Elamur Research⌕
☰
home / papers / 2024 / Multi-Label Classification for Implicit Discourse Relation Recognition

Multi-Label Classification for Implicit Discourse Relation Recognition

Summary

In PDTB-3, annotators sometimes give one implicit discourse relation two senses at once, such as a temporal and a concessive reading of the same argument pair. Earlier implicit discourse relation recognition (IDRR) systems split such cases into separate training items and counted a prediction as correct if it hit either gold sense. This ignores how co-occurring senses relate and cannot tell single-sense from multi-sense cases. The authors reframe IDRR as multi-label classification, which they describe as a first for this task. They compare three RoBERTa-based models:

  • a single sigmoid head,
  • separate per-label binary softmax heads,
  • a GRU decoder that generates labels in sequence.

To cope with rare senses and sense pairs, they evaluate on Level-2 senses with section-level cross-validation instead of the usual fixed split. Multi-label training does not hurt single-label prediction and can slightly improve it, with the per-label-head model performing best. The paper also analyses errors and data sparsity, and argues that richer, more varied multi-sense corpora are needed.

Key points

  • Data: implicit relations in PDTB-3, both intra- and inter-sentential. Roughly 5% of them carry two sense labels.
  • Evaluation uses 12 section-level folds (21 sections train, 2 dev, 2 test) and macro-averaged F1. Precision, Recall and Hamming loss are given in the appendix.
  • Methods:
    • Method 1: one sigmoid BCE head on [CLS].
    • Method 2: per-label binary softmax heads.
    • Method 3: GRU decoder with dot-product attention and beam search (beam size 4).
    • All use RoBERTa with learning rate 1e−5 and batch size 64.
  • Method 2 scores best, around 53.1 total macro-F1 versus about 52.4 for Method 1. The authors also report focal loss as a preferable loss for this task.
  • Cause, Condition, Purpose and Conjunction are recognised reliably across methods. Cause+Belief (near 0 F1), Synchronous and Equivalence remain hard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-acl.500 · arXiv 2406.04461

@inproceedings{long2024multi,
  title = {{Multi-Label Classification for Implicit Discourse Relation Recognition}},
  author = {Long, Wanqiu and Siddharth, N. and Webber, Bonnie},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.500},
  eprint = {2406.04461},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.500}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10704– 10716, Abu Dhabi, United Arab Emirates. Associ… find our card
  2. Hassan Alhuzali and Sophia Ananiadou. 2021. SpanEmo: Casting multi-label emotion classification as span-prediction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1573–1584, Online. Association for Computational Ling… find
  3. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571–583, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  4. William C. Mann, Christian M.I.M. Matthiessen, and Sandra A. Thompson. 1989. Rhetorical structure theory and text analysis. find
  5. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval: Scoring, term weighting, and the vector space model. find
  6. Yin Jou Huang and Sadao Kurohashi. 2021. Extractive summarization considering discourse and coreference relations based on heterogeneous graph. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3046–3052, Online. Associ… find our card
  7. Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3659–3668, Gyeongju, Republic of Korea. International Committee on Computational L… find
  8. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  9. Yawen Ouyang, Zhanghua Wu, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2022. Towards multi-label unknown intent detection. In International Conference on Computational Linguistics. find
  10. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414, Online. Association for Computational Lingui… find our card
  11. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  12. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  13. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819, Online. Association for C… find our card
  14. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 49(1):157–198. find our card
  15. Nankai Lin, Guanqiu Qin, Gang Wang, Dong Zhou, and Aimin Yang. 2023. An effective deployment of contrastive learning in multi-label text classification. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8730–8744, Toronto, Canada. Association for Computational Linguistics… find
  16. Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999–3007. find our card
  17. Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257–2267, Melbourne, Australia. Asso… find our card
  18. Robert E. Schapire and Yoram Singer. 1998. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37:297–336. find
  19. Merel Scholman, Tianai Dong, Frances Yung, and Vera Demberg. 2022a. DiscoGeM: A crowdsourced corpus of genre-mixed implicit discourse relations. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3281–3290, Marseille, France. European Language Resources Association. find our card
  20. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911, Gyeongju, Republic of Korea. International Committee on Co… find our card
  21. Merel Scholman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, and Vera Demberg. 2022b. Design choices in crowdsourcing discourse relation annotations: The effect of worker selection and training. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2148–2… find our card
  22. Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL… find
  23. Merel C. J. Scholman and Vera Demberg. 2017. Examples and specifications that prove a point: Identifying elaborative and argumentative discourse relations. Dialogue Discourse, 8:56–83. find our card
  24. Ramil Yarullin and Pavel Serdyukov. 2021. BERT for Sequence-to-Sequence Multi-label Text Classification, pages 187–198. find
  25. Wei Shi and Vera Demberg. 2017. On the need of cross validation for discourse relation classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 150–156, Valencia, Spain. Association for Computational… find our card
  26. Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1672–1681, Dublin, Ireland. Association for Computational Linguistics. find
  27. Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xi-anpei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. find
  28. Che-Ping Tsai and Hung yi Lee. 2019. Order-free learning alleviating exposure bias in multi-label classification. In AAAI Conference on Artificial Intelligence. find
  29. Cheng Wang, Jorge Balazs, György Szarvas, Patrick Ernst, Lahari Poddar, and Pavel Danchenko. 2022. Calibrating imbalanced classifiers with focal loss: An empirical study. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 145–153, Abu Dha… find
  30. Jianfei Yu, Luís Marujo, Jing Jiang, Pradeep Karuturi, and William Brendel. 2018. Improving multi-label emotion classification via sentiment classification with dual attention transfer network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1097–1102… find
  31. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. find
  32. Ximing Zhang, Qian-Wen Zhang, Zhao Yan, Ruifang Liu, and Yunbo Cao. 2021a. Enhancing label correlation feedback in multi-label text classification via multi-task learning. In Findings. find
  33. Yingxue Zhang, Fandong Meng, Peng Li, Ping Jian, and Jie Zhou. 2021b. Context tracking network: Graph-based context modeling for implicit discourse relation recognition. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  34. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. find
  35. Hongyi Wu, Hao Zhou, Man Lan, Yuanbin Wu, and Yadong Zhang. 2023. Connective prediction for implicit discourse relation recognition via knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5908–5923, Toronto… find our card
  36. Comparison Contingency Expansion Temporal Comparison.Concession Comparison.Contrast Contingency.Cause Contingency.Cause+Belief Contingency.Condition Contingency.Purpose Expansion.Conjunction Expansion.Equivalence Expansion.Instantiation Expansion.Level-of-detail Expansion.Manner Expansion.Substituti… find
  37. Asynchronous/Cause Asynchronous/Contrast Cause+Belief/Instantiation Cause+Belief/Level-of-detail Cause/Asynchronous Cause/Instantiation Cause/Level-of-detail Cause/Manner Cause/Substitution Concession/Asynchronous Concession/Substitution Condition/Manner Conjunction/Asynchronous Conjunction/Cause Co… find

↑ back to top

This page in Markdown →