Elamur Research⌕
☰
home / papers / 2023 / Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition

Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition

Summary

Implicit discourse relation recognition (IDRR) assigns a sense to a pair of arguments that are not linked by an explicit connective. The PDTB annotates these senses in a three-level hierarchy. Earlier hierarchy-aware models used only the full sense tree, which the authors call the global hierarchy. They did not exploit the label path attached to each example, which the authors call the local hierarchy.

GOLF combines three parts:

  • a RoBERTa-based encoder with Multi-Head Interactive Attention (MHIA) between the two arguments;
  • a top-down "staircase" classifier, in which predictions at higher levels condition the levels below;
  • two contrastive objectives used only during training.

The global objective builds sense embeddings with a GCN over the hierarchy. It then pulls each relation representation toward its gold labels in a shared text–label space. The local objective uses a new scoring function that grades how similar two examples' sense sequences are, and that score controls how close their representations should be.

On PDTB 2.0 and PDTB 3.0, GOLF beats prior state-of-the-art systems at every level. Its predictions across levels are also more consistent. This shows that structured label information can improve the quality of discourse representations.

Key points

  • Data and setup: experiments use PDTB 2.0 and PDTB 3.0 with top-level, second-level and connective-level senses. There are two variants, GOLF (base) and GOLF (large, RoBERTa-large). The code is public at github.com/YJiangcm/GOLF_for_IDRR.

  • Multi-level consistency (Top-Sec / Top-Sec-Conn), GOLF (large) vs. the previous best model LDSGM:

    DatasetGOLF (large)LDSGM
    PDTB 2.061.79 / 36.0058.61 / 26.85
    PDTB 3.064.86 / 38.2660.32 / 34.57

  • Rare senses: the gains concentrate on infrequent labels. On PDTB 2.0, GOLF (base) raises Exp.List F1 from 8.98 to 27.78. GOLF (large) is the first among the compared models to score above zero on Temp.Synchrony (13.33) and Comp.Concession (11.11).

  • Ablation:

    • Removing any module (MHIA, staircase, global loss or local loss) lowers results at every level.
    • The local contrastive loss contributes the most.
    • Replacing the graded similarity score with a hard-label version also degrades performance.

  • Inference: the contrastive modules are dropped at test time, so only the encoder and the classifier are run.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.510 · arXiv 2211.13873

@inproceedings{jiang2023global,
  title = {{Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition}},
  author = {Jiang, Yuxin and Zhang, Linhan and Wang, Wei},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.510},
  eprint = {2211.13873},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.510}
}

References

47 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In this section, we illustrate the limitations of our method, which could be summarized into the following two aspects. Firstly, since the cumbersome data annotation leads to few publicly available datasets of IDRR tasks, we only conduct experiments on English corpora including PDTB 2.0 and PDTB 3.0… find
  2. Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. 2021. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26989–27002. \#\# Ethics Statement Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan… find our card
  3. Since our method relies on pre-trained language models, it may run the danger of inheriting and propagating some of the models’ negative biases from the data they have been pre-trained on (Bender et al., 2021). Furthermore, we do not see any other potential risks. \#\# Acknowledgments W. Wang was supp… find
  4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.040… find our card
  5. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610… find
  6. Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023a. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. CoRR, abs/2304.14827. find our card
  7. Boli Chen, Xin Huang, Lin Xiao, Zixin Cai, and Liping Jing. 2020a. Hyperbolic interaction model for hierarchical multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7496–7503. find
  8. Tao Chen, Haizhou Shi, Siliang Tang, Zhigang Chen, Fei Wu, and Yueting Zhuang. 2021. CIL: contrastive instance learning framework for distantly supervised relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  9. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020b. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine… find our card
  10. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  11. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  12. Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find our card
  13. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. CoRR, abs/2305.12870. — [arXiv:2305.12870](https://arxiv.org/abs/2305.12870) find
  14. Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3021–3035, Abu Dhabi, United Arab Emirates. Association for Computat… find
  15. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 1152– 1158. Euro… find our card
  16. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  17. Maria Liakata, Simon Dobnik, Shyamasree Saha, Colin Batchelor, and Dietrich Rebholz Schuhmann. 2013. A discourse-driven content model for summarising scientific articles evaluated in a complex question answering task. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pro… find
  18. Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find our card
  19. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3830–3836. ijcai.org. find
  20. Fengyu Guo, Ruifang He, Jianwu Dang, and Jian Wang. 2020. Working memory-driven neural networks with a novel knowledge enhancement paradigm for implicit discourse relation recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7822–7829. find
  21. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738. find our card
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034. find
  23. Wu Hong, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. 2022. Sentence-aware contrastive learning for open-domain passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1062–1074. find
  24. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Trans. Assoc. Comput. Linguistics, 3:329–344. find our card
  25. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1224–1233. The… find our card
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  27. Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11,… find our card
  28. Linh The Nguyen, Ngo Van Linh, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence,… find our card
  29. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  30. TB OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,… find
  32. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112. find our card
  33. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. find our card
  34. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). find our card
  35. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow discourse parsing using convolutional neural network. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning: Shared Task, CoNLL 2016, Berlin, Germany, August 7-12, 2016, pages 70–77. ACL. find our card
  36. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-… find our card
  37. Wei Shi and Vera Demberg. 2019. Learning to explic-itate connectives with seq2seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics, IWCS 2019, Long Papers, Gothenburg, Sweden, May 23-27 May, 2019, pages 188–199. As… find
  38. Ziyi Shou, Yuxin Jiang, and Fangzhen Lin. 2022. AMR-DA: Data augmentation by Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3082–3098, Dublin, Ireland. Association for Computational Linguistics. find
  39. Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xi-anpei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  40. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania, 35:108. find
  41. Max Welling and Thomas N Kipf. 2016. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017). find our card
  42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language p… find our card
  43. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11486–11494. find
  44. Changxing Wu, Chaowen Hu, Ruochen Li, Hongyu Lin, and Jinsong Su. 2020. Hierarchical multi-task learning with CRF for implicit discourse relation recognition. Knowl. Based Syst., 195:105637. find
  45. Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022a. Encoding and fusing semantic connection and linguistic evidence for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3247–3257, Dublin, Ireland. Association for Computational Ling… find our card
  46. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022b. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911, Gyeongju, Republic of Korea. International Committee on C… find our card
  47. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2023-global-local-hierarchy-aware-contrastive/