Elamur Research⌕
☰
home / papers / 2022 / Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations

Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations

Summary

The paper addresses implicit discourse relation recognition: deciding which sense links two adjacent text spans when no explicit connective marks it. PDTB-2 and PDTB-3 organize senses into a three-level hierarchy. Earlier systems mostly used this hierarchy only to choose which labels to predict. The authors instead use it to guide contrastive learning. Their model is a multi-task RoBERTa-base system that predicts level-1 and level-2 senses from the [CLS] representation of the concatenated arguments. A supervised contrastive loss pulls together instances of the same level-2/level-3 type and pushes apart instances of sister types, with negatives chosen by their position in the hierarchy. To keep enough positives per batch, they enlarge the training data with the explicit connectives that PDTB annotators recorded for each implicit relation, inserting each connective at the start of Arg2 as a new example. The approach reaches state-of-the-art results on both corpora and handles minority labels better than earlier systems. This shows that the structure of the label inventory is useful for learning, not just for defining targets.

Key points

  • Evaluated on both PDTB-2 and PDTB-3, which is less common than using a single benchmark. Code is available at github.com/wanqiulong0923/Contrastive_IDRR.
  • Full model results:
    • PDTB-2: top-level Acc/Macro-F1 of 72.18/69.60 and second-level 61.69/49.66.
    • PDTB-3: top-level 75.31/70.05 and second-level 64.68/57.62.
  • Gains over a plain RoBERTa baseline and a multi-task variant (RoBERTa-MTL). For example, PDTB-3 second-level accuracy rises from 60.56 to 64.68.
  • Removing connective-based data augmentation clearly hurts. Second-level Macro-F1 falls to 45.54 on PDTB-2 and 51.80 on PDTB-3. The authors attribute this to the many level-2 classes (11 in PDTB-2, 14 in PDTB-3) leaving too few positives in batches of 256.
  • Comparing negative-selection strategies showed that treating all examples from other level-1 classes as negatives can harm the learned representations. Limitations:
    • Multi-label instances are split into separate examples.
    • Experiments cover English only.
    • Extension to other discourse banks (e.g., TED-MDB) is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

27 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 17 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.734 · arXiv 2301.02724

@inproceedings{long2022facilitating,
  title = {{Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations}},
  author = {Long, Wanqiu and Webber, Bonnie},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.734},
  eprint = {2301.02724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.734}
}

References

57 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josephine Sarpong Akosa. 2017. Predictive accuracy : A misleading performance measure for highly imbalanced data. find
  2. Moustafa Farid Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In EMNLP. find our card
  3. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571–583, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  4. Mohamed Bekkar, Hassiba Kheliouane Djemaa, and Taklit Akrouf Alitouche. 2013. Evaluation measures for models assessment over imbalanced data sets. Journal of Information Engineering and Applications, 3:27–38. find
  5. Rui Cao, Yihao Wang, Yu Guo Liang, Ling Gao, Jie Zheng, Jie Ren, and Zheng Wang. 2022. Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding. In FINDINGS. find
  6. Jifan Chen, Qi Zhang, Pengfei Liu, and Xuanjing Huang. 2016a. Discourse relations detection via a mixed generative-discriminative framework. In AAAI. find
  7. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016b. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1726–1735, B… find our card
  8. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR. find our card
  9. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  10. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  11. Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  13. Zujun Dou, Yu Hong, Yu Sun, and Guodong Zhou. 2021. CVAE-based re-anchoring for implicit discourse relation classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1275–1283, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  14. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistic… find our card
  15. Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations. find our card
  16. Fengyu Guo, Ruifang He, Jianwu Dang, and Jian Wang. 2020. Working memory-driven neural networks with a novel knowledge enhancement paradigm for implicit discourse relation recognition. In AAAI. find
  17. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  18. Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recognizing implicit discourse relation on joint modeling of classification and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2418–2431, Onl… find our card
  19. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661–18673. Curran Associates, Inc. find
  20. Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for BERT sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long P… find
  21. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). find
  22. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting bert to implicit discourse relation classification with a focus on discourse connectives. In LREC. find our card
  23. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299–1308, Cop… find our card
  24. Li Liang, Zheng Zhao, and Bonnie Webber. 2020. Extending implicit discourse relation recognition to the PDTB-3. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 135– 147, Online. Association for Computational Linguistics. find our card
  25. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3830–3836. International J… find
  26. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224–1233, Austin, Texas. Association for Computational Linguistic… find our card
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  28. Wanqiu Long, Bonnie Lynn Webber, and Deyi Xiong. 2020. Ted-cdb: A large-scale chinese discourse relation dataset on ted talks. In EMNLP. find our card
  29. Thomas Meyer, Najeh Hajlaoui, and Andrei Popescu-Belis. 2015. Disambiguating discourse connectives for statistical machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23:1–1. find
  30. Jirí Mírovský, Pavlína Jínová, and Lucie Poláková. 2014. Discourse relations in the prague dependency treebank 3.0. In COLING. find
  31. Marc Moens and Mark Steedman. 1988. Temporal ontology and temporal reference. Computational Linguistics, 14(2):15–28. find
  32. Linh The Nguyen, Ngo Van Linh, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In ACL. find our card
  33. Silvia Pareti and Irina Prodanof. 2010. Annotating attribution relations: Towards an Italian discourse treebank. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  34. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In EMNLP. find our card
  35. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  36. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  37. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow discourse parsing using convolutional neural network. In Proceedings of the CoNLL-16 shared task, pages 70–77, Berlin, Germany. Association for Computational Linguistics. find our card
  38. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006–1017, Vanco… find our card
  39. Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. International Conference on Learning Representations. find
  40. Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, and Min Zhang. 2020. Interactively-propagative attention learning for implicit discourse relation recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3168–3178, Barcelona, Spain (Online). Intern… find our card
  41. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7… find our card
  42. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823. find our card
  43. Wei Shi and Vera Demberg. 2019a. Learning to explici-tate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden. Association for Computational Li… find
  44. Wei Shi and Vera Demberg. 2019b. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E… find our card
  45. Varsha Suresh and Desmond C. Ong. 2021. Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification. In EMNLP. find
  46. Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xi-anpei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  47. Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021. CLINE: Contrastive learning with semantic negative examples for natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan… find
  48. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020b. Clear: Contrastive learning for sentence representation. ArXiv, abs/2012.15466. — [arXiv:2012.15466](https://arxiv.org/abs/2012.15466) find
  49. Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022. Encoding and fusing semantic connection and linguistic evidence for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3247–3257, Dublin, Ireland. Association for Computational Lingu… find our card
  50. Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… find our card
  51. Seonghyeon Ye, Jiseon Kim, and Alice Oh. 2021. Efficient contrastive learning via novel data augmentation and curriculum learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1832–1838, Online and Punta Cana, Dominican Republic. Association for Co… find
  52. Deniz Zeyrek and Murathan Kurfalı. 2017. TDB 1.1: Extensions on Turkish discourse bank. In Proceedings of the 11th Linguistic Annotation Workshop, pages 76–81, Valencia, Spain. Association for Computational Linguistics. find
  53. Deniz Zeyrek, Amalia Mendes, Yulia Grishina, Murathan Kurfali, Samuel Gibbon, and Maciej Ogrodniczuk. 2019. Ted multilingual discourse bank (ted-mdb): a parallel corpus annotated in the pdtb style. Language Resources and Evaluation, pages 1–38. find
  54. Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021. Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5786–5798, Online and Pu… find
  55. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. find
  56. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. In AAAI. find
  57. Changxing Wu, Chaowen Hu, Ruochen Li, Hongyu Lin, and Jinsong Su. 2020a. Hierarchical multi-task learning with crf for implicit discourse relation recognition. Knowl. Based Syst., 195:105637. find

↑ back to top