Elamur Research⌕
☰
home / papers / 2024 / Exploring Soft-Label Training for Implicit Discourse Relation Recognition

Exploring Soft-Label Training for Implicit Discourse Relation Recognition

Summary

Implicit discourse relation recognition (IDRR) assigns a PDTB sense to two adjacent text spans that have no explicit connective. It is difficult partly because readers often disagree about which relation is intended. Instead of forcing a single gold label, Costa and Kosseim fine-tune RoBERTa-base on the full crowdsourced sense distributions of the DiscoGeM corpus using a soft cross-entropy loss. At test time, the top-scoring output gives one PDTB 3.0 third-level sense, which is compared with the majority label. The 6,505 relations were split 80/20, with the split stratified by majority sense. Epochs, batch size, learning rate and decay were tuned. The best configuration reaches an F1 of 51.38 at the third level. Input with only the two arguments beats input with surrounding context, which the authors attribute to truncation. The study is an early test of learning from annotator disagreement in discourse parsing. However, it reports no hard-label baseline and no PDTB 3.0 evaluation, and tuning results are reported on the test split. The advantage of soft labels is therefore suggested rather than demonstrated.

Key points

  • Data: DiscoGeM contains 6,505 inter-sentential implicit relations.
    • By genre: 2,800 political, 3,060 literary and 645 encyclopedic.
    • It has 65,863 crowd annotations from 199 participants.
    • Only 27 of the 36 PDTB 3.0 third-level senses are covered; BELIEF and SPEECH ACT are excluded.
    • The split is 5,204 training and 1,301 test relations.
  • Model: RoBERTa-base (Hugging Face) with one classification layer, trained with soft cross-entropy and Adam (β1 = 0.9, β2 = 0.98). The best setup used 10 epochs, batch size 64, learning rate 1e-5 and no decay.
  • Results: With argument-only input (ARG1+ARG2), the model reaches F1 51.38, precision 51.54 and recall 52.19. Adding context (average 531 vs 245 characters) drops F1 to 43.67.
  • Context for the numbers: Comparisons with other systems are indirect, since they use different corpora, sense levels or metrics. Liu and Strube (2023) report F1 55.26 on PDTB 3.0 at the second level. Pyatkin et al. (2023) report 41% accuracy.
  • Code: GitHub, CLaC-Lab/Implicit-Discourse-Relation-Recognition. Planned future work includes soft-label evaluation metrics, testing on PDTB 3.0 and a single-label baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.codi-1.11

@inproceedings{costa2024exploring,
  title = {{Exploring Soft-Label Training for Implicit Discourse Relation Recognition}},
  author = {Costa, Nelson Filipe and Kosseim, Leila},
  booktitle = {Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.codi-1.11},
  url = {https://doi.org/10.18653/v1/2024.codi-1.11}
}

References

25 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lora Aroyo and Chris Welty. 2013. Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. In Proceedings of the 5th Annual Association for Computing Machinery Web Science Conference (WebSci’13), Paris, France. Association for Computing Machinery (ACM). find
  2. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We Need to Consider Disagreement in Evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computa… find our card
  3. Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Y Wong, and Simon See. 2023. DiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL’23), pages 35–57, T… find our card
  4. Nelson Filipe Costa, Nadia Sheikh, and Leila Kosseim. 2023. Mapping Explicit and Implicit Discourse Relations between the RST-DT and the PDTB 3.0. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing (RANLP’23), pages 344–352, Varna, Bulgaria. find our card
  5. Vera Demberg, Merel CJ Scholman, and Fatemeh Torabi Asr. 2019. How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse, 10(1):87–135. find
  6. Jet Hoek, Merel C.J. Scholman, and Ted J.M. Sanders. 2021. Is there less annotator agreement when the discourse relation is underspecified? In Proceedings of the 1st Workshop on Integrating Perspectives on Discourse Annotation, pages 1–6, Tübingen, Germany. Association for Computational Linguistics… find our card
  7. Patrick Huber, Wen Xiao, and Giuseppe Carenini. 2021. W-RST: Towards a Weighted RST-style Discourse Framework. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics Wei Liu and Michael Strube. 2023. Annotation-Inspired Implicit Discourse Relation Classification w… find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  9. Wanqiu Long and Bonnie Webber. 2022. Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP’22), pages 10704–10716, Abu Dhabi, United Arab Emirat… find our card
  10. William C Mann and Sandra A Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  11. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), pages 2237–2240, Lisbon, Portugal. European Language Resources Association (ELRA). find our card
  12. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics (TACL), 7:677–694. find our card
  13. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pages 2961–2968, Marrakech, Morocco. European Langua… find our card
  14. Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. LDC2019T05. Web Download. Philadelphia: Linguistic Data Consortium. find
  15. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20), pages 2804–2819, Online. Association fo… find our card
  16. Haodong Zhao, Ruifang He, Mengnan Xiao, and Jing Xu. 2023. Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL’23),… find our card
  17. Valentina Pyatkin, Frances Yung, Merel C. J. Scholman, Reut Tsarfaty, Ido Dagan, and Vera Demberg. 2023. Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design. Transactions of the Association for Computational Linguistics (TACL), 11:1014– 1032. find our card
  18. Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher N. L. Clark, Annie Louis, and Bonnie Webber. 2016. Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict, and Context-Dependence in a Crowdsourced Elicitation Task. In Proceedings of the 10th Linguistic Annotati… find
  19. Merel Scholman, Tianai Dong, Frances Yung, and Vera Demberg. 2022a. DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC’22), pages 3281– 3290, Marseille, France. European Language Resources A… find our card
  20. Merel Scholman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, and Vera Demberg. 2022b. Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC’22), pa… find our card
  21. Manfred Stede. 2008. Disambiguating Rhetorical Structure. Research on Language and Computation, 6(3):311–332. find
  22. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  23. CoNLL 2016 Shared Task on Multilingual Shallow Discourse Parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19, Berlin, Germany. Association for Computational Linguistics (ACL). find our card
  24. Frances Yung, Vera Demberg, and Merel Scholman. 2019. Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task. In Proceedings of the 13th Linguistic Annotation Workshop, pages 16–25, Florence, Italy. Association for Computational Linguistics (ACL). find our card
  25. R EASON R ESULT ARG1- AS -C OND ARG2- AS -C OND ARG1- AS -N EG C OND find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/costa-2024-exploring-soft-label-training-implicit/