Elamur Research⌕
☰
home / papers / 2023 / Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation

Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation

Summary

Implicit discourse relations are hard to classify because no connective (such as "but" or "because") signals the link between two arguments. Earlier work used annotated implicit connectives only as an extra training signal, which gave little benefit. Liu and Strube instead mimic how PDTB annotators work: they first insert a suitable connective and then choose a relation label. Their end-to-end model is built on RoBERTa. It predicts a connective at a [MASK] position between Arg1 and Arg2, then feeds the arguments together with that connective into a relation classifier. Both modules share the embeddings and Transformer layers. Connectives generated early in training are unreliable, and gold connectives are unavailable at test time. To handle both issues, the authors apply Scheduled Sampling, which moves the classifier gradually from annotated to generated connectives. On PDTB 2.0, PDTB 3.0 and the German PCC, the model beats other connective-enhanced baselines and comes close to state-of-the-art systems. The work shows that placing connectives directly in the input is more effective than using them only as auxiliary targets.

Key points

  • Level-1 results (Ji split): On PDTB 2.0 the model reaches 74.59 accuracy and 68.64 F1, against 68.61 and 60.89 for plain RoBERTa. On PDTB 3.0 it scores 76.23 and 71.15, against 73.51 and 67.98.
  • Ablation: Removing Scheduled Sampling lowers PDTB 2.0 F1 to 66.68. Also dropping the connective-generation loss lowers it further to 63.43, which still beats RoBERTa. This suggests the annotation-inspired architecture helps on its own.
  • Per-relation results: Contingency.Cause improves most (F1 66.35 vs. 60.03). However, all models fail on rare senses such as Temporal.Synchrony, Contingency.Pragmatic cause and Comparison.Concession.
  • Analysis findings:
    • End-to-end training makes the model more robust to wrongly generated connectives.
    • Incorrect connectives remain the main bottleneck.
    • The model balances evidence from the arguments and the connective better than the baselines do.
  • Practical notes: Parameter count and memory use are close to RoBERTa, but training is slower because each input passes through the encoder twice. Multi-label instances (3.45% in PDTB 2.0, 5% in PDTB 3.0) are reduced to their first label. The code is released as ConnRel on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.874 · arXiv 2306.06480

@inproceedings{liu2023annotation,
  title = {{Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation}},
  author = {Liu, Wei and Strube, Michael},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.874},
  eprint = {2306.06480},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.874}
}

References

52 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, volume 28. find
  2. Peter Bourgonje. 2021. Shallow Discourse Parsing for German. Doctoral Thesis, Universität Potsdam. find
  3. Peter Bourgonje and Manfred Stede. 2020. The Potsdam commentary corpus 2.2: Extending annotations for shallow discourse parsing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1061–1066, Marseille, France. European Language Resources Association. find our card
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597–1607. PMLR. find our card
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Long-biao Wang, and Xiangang Li. 2018. Implicit discourse relation recognition using neural tensor network with interactive attention and sparse learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 547–558, San… find our card
  8. Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125–179. find our card
  9. Freya Hewett, Roshan Prakash Rane, Nina Harlacher, and Manfred Stede. 2019. The utility of discourse parsing features for predicting argumentation structure. In Proceedings of the 6th Workshop on Argument Mining, pages 98–103, Florence, Italy. Association for Computational Linguistics. find our card
  10. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017. find our card
  11. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  12. Congcong Jiang, Tieyun Qian, Zhuang Chen, Kejian Tang, Shaohui Zhan, and Tao Zhan. 2021. Generating pseudo connectives with mlms for implicit discourse relation recognition. In PRICAI 2021: Trends in Artificial Intelligence, pages 113–126, Cham. Springer International Publishing. find
  13. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414, Online. Association for Computational Lingui… find our card
  14. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2018. A knowledge-augmented neural network model for implicit discourse relation classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 584–595, Santa Fe, New Mexico, USA. Association for Computatio… find
  15. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1152– 1158, Marseille, France. European Language Resources A… find our card
  16. Murathan Kurfalı and Robert Östling. 2021. Let’s be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction. In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language, pages 1–10, Online. Association for Computa… find our card
  17. Wenqiang Lei, Xuancong Wang, Meichun Liu, Ilija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: A simple word interaction model for implicit discourse relation recognition. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 4026–4032. find
  18. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 343–351, Singapore. Association for Computational Linguistics. find our card
  19. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 997–1006, Portland, Oregon, USA. Association for Computati… find our card
  20. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  21. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3830–3836. International J… find
  22. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224–1233, Austin, Texas. Association for Computational Linguistic… find our card
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT Pretraining Approach. CoRR, abs/1907.11692. find
  24. Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10704– 10716, Abu Dhabi, United Arab Emirates. Associ… find our card
  25. Wang Longyue. 2019. Discourse-aware neural machine translation. Ph.D. thesis, Dublin City University. find
  26. Thomas Meyer. 2015. Discourse-level features for statistical machine translation. Ph.D. thesis, École polytechnique fédérale de Lausanne (EPFL). find
  27. Todor Mihaylov and Anette Frank. 2019. Discourse-aware semantic self-attention for narrative reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), page… find
  28. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105–1116, Berlin, Germany. Association for Computational Linguis… find our card
  29. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–691, S… find our card
  30. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16, Suntec, Singapore. Association for Computational Linguistics. find our card
  31. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  32. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006–1017, Vanco… find our card
  33. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016. find our card
  34. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 645–654, Gothenburg, Sweden. As… find our card
  35. Jihao Shi, Xiao Ding, Li Du, Ting Liu, and Bing Qin. 2021. Neural natural logic inference for interpretable question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3673–3684, Online and Punta Cana, Dominican Republic. Association for Compu… find
  36. Wei Shi and Vera Demberg. 2019. Learning to explici-tate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden. Association for Computational Lin… find
  37. Wei Shi, Frances Yung, Raphael Rubino, and Vera Demberg. 2017. Using explicit discourse connectives in translation for implicit discourse relation classification. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 484–495, Taipei… find our card
  38. Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  39. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, Berlin, Germany. Association for Computational Linguis… find our card
  40. Caroline Sporleder and Alex Lascarides. 2008. Using automatically labelled examples to classify rhetorical relations: an assessment. Natural Language Engineering, 14(3):369–416. find
  41. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038, Brussels, Belgium. Association for Comput… find our card
  42. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911, Gyeongju, Republic of Korea. International Committee on Co… find our card
  43. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  44. CoNLL 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19, Berlin, Germany. Association for Computational Linguistics. find our card
  45. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Lisbon, Portugal. Association… find our card
  46. Yue Zhang. 2018. Joint models for NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, Melbourne, Australia. Association for Computational Linguistics. find
  47. Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3848–3858, Abu Dhabi, United Arab Emirates. Associati… find our card
  48. Zhi Min Zhou, Man Lan, Zheng Yu Niu, Yu Xu, and Jian Su. 2010. The effects of discourse connectives prediction on implicit discourse relation recognition. In Proceedings of the SIGDIAL 2010 Conference, pages 139–146, Tokyo, Japan. Association for Computational Linguistics. find our card
  49. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The Penn Discourse TreeBank 3.0 annotation manual. Philadelphia, University of Pennsylvania, 35:108. find
  50. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11486–11494. find
  51. Wei Xiang and Bang Wang. 2023. A survey of implicit discourse relation recognition. ACM Computing Surveys, 55(12):1–34. find
  52. L1 L2 PDTB 2.0 Comparison Contingency Expansion Temporal Comparison.Concession Comparison.Contrast Contingency.Cause Contingency.Pragmatic cause Expansion.Conjunction Expansion.Instantiation Expansion.Alternative Expansion.List Expansion.Restatement Temporal.Asynchronous Temporal.Synchrony PDTB 3.0… find

↑ back to top

This page in Markdown →