Elamur Research⌕
☰
home / papers / 2023 / Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation

Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation

Summary

Implicit discourse relation recognition (IDRR) asks a model to name the semantic relation between two arguments when no connective is written in the text, which makes it much harder than the explicit case. Earlier prompt-based approaches have a PLM fill in a connective, but they train against one-hot targets. Those targets treat near-synonymous connectives as wrong answers, and the approaches also rely on carefully engineered templates. The authors propose CP-KD, which pairs connective prediction with knowledge distillation. A RoBERTa teacher sees the arguments in a minimal "Arg1 [MASK] Arg2" template plus an appended hint naming the gold sense. Its temperature-smoothed connective distribution then acts as a soft target for a student that receives no hint, trained with a KL-divergence loss. Predicted connectives are mapped to sense labels. Evaluated on PDTB 2.0, PDTB 3.0 and CoNLL 2016, the method reportedly beats prior state of the art by roughly 3% on both coarse and fine senses. It holds up across different templates and transfers to explicit relations, which shows that softened connective labels capture useful relation knowledge.

Key points

  • The teacher input appends an answer hint ("Answer: sense word") to the masked-connective template, and the student learns from the teacher's soft labels instead of relying only on hard one-hot connectives.
  • On PDTB 3.0 top-level senses with the simple template, CP-KD-base scores 72.07 F1 / 77.00 Acc versus ConnPrompt's 69.91 / 74.36. With multi-prompt ensembling it reaches 72.89 / 77.54.
  • On PDTB 2.0 top-level senses with the PCP template, CP-KD-large gets 71.37 F1 / 76.58 Acc against 67.79 / 73.80 for PCP-large.
  • The explicit variant KD-EDRR adds connective prediction as an auxiliary task and achieves 96.39 Acc / 95.59 F1 on PDTB 2.0 top-level EDRR, a new best in the reported comparison.
  • One limitation is that only single-token connectives can serve as answer words. Code is available at github.com/cubenlp/CP_KD-for-IDRR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.325

@inproceedings{wu2023connective,
  title = {{Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation}},
  author = {Wu, Hongyi and Zhou, Hao and Lan, Man and Wu, Yuanbin and Zhang, Yadong},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.325},
  url = {https://doi.org/10.18653/v1/2023.acl-long.325}
}

References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fatemeh Torabi Asr and Vera Demberg. 2020. Interpretation of discourse connectives is probabilistic: Evidence from the study of but and although. Discourse Processes, 57(4):376–399. find
  2. International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2976–2987. find
  3. Gongfan Fang, Yifan Bao, Jie Song, Xinchao Wang, Donglin Xie, Chengchao Shen, and Mingli Song. 2021. Mosaicking to distill: Knowledge distillation from out-of-domain data. Advances in Neural Information Processing Systems, 34:11920–11932. find
  4. John R Hershey and Peder A Olsen. 2007. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, volume 4, pages IV–317. find
  5. Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7). find our card
  6. Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. 2021. Dagn: Discourse-aware graph network for logical reasoning. arXiv e-prints, pages arXiv–2103. find
  7. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  8. Congcong Jiang, Tieyun Qian, and Bing Liu. 2022. Knowledge distillation for discourse relation analysis. In Companion Proceedings of the Web Conference 2022, WWW ’22, page 210–214, New York, NY, USA. Association for Computing Machinery. find
  9. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1152– 1158. find our card
  10. Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. 2020. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12925–12935. find
  11. Murathan Kurfalı and Robert Östling. 2021. Let’s be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction. arXiv preprint arXiv:2106.03192. find our card
  12. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4793–4801. IEEE. find
  13. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299–1308. find our card
  14. Dongha Choi, HongSeok Choi, and Hyunju Lee. 2022. Domain knowledge transferring for pre-trained language model via calibrated activation boundary distillation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1658–1669. find
  15. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, pages 5958–5968. find
  16. Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009.… find our card
  17. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9). find our card
  18. Siddharth Varia, Christopher Hidey, and Tuhin Chakrabarty. 2019. Discourse relation prediction: Revisiting word pairs with convolutional networks. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 442–452. find our card
  19. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2021. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20. find
  20. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for English and Chinese in CoNLL-2016 shared task. In Proceedings of the CoNLL-16 shared task, pages 33–40. find our card
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  22. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  23. Linh The Nguyen, Linh Van Ngo, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4201–4207. find our card
  24. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16, Suntec, Singapore. Association for Computational Linguistics. find our card
  25. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). find our card
  26. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352. find our card
  27. Ieva Staliunaite, Philip John Gorinski, and Ignacio Iacobacci. 2021. Improving commonsense causal reasoning by adversarial training and data augmentation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intellige… find
  28. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania, 35:108. find
  29. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative App… find
  30. Changxing Wu, Chaowen Hu, Ruochen Li, Hongyu Lin, and Jinsong Su. 2020. Hierarchical multi-task learning with crf for implicit discourse relation recognition. Knowledge-Based Systems, 195:105637. find
  31. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911. find our card
  32. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  33. CoNLL 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19. find our card
  34. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2019. Model compression with multi-task knowledge distillation for web-scale question answering system. arXiv preprint arXiv:1904.09636. — [arXiv:1904.09636](https://arxiv.org/abs/1904.09636) find
  35. Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. arXiv preprint arXiv:2210.07032. find our card
  36. Zhi Min Zhou, Man Lan, Zheng Yu Niu, Yu Xu, and Jian Su. 2010. The effects of discourse connectives prediction on implicit discourse relation recognition. In Proceedings of the SIGDIAL 2010 Conference, pages 139–146. find our card
  37. Zhiyuan Tang, Dong Wang, and Zhiyong Zhang. 2016. Recurrent neural network training with dark knowledge transfer. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5900–5904. find
  38. A The CoNLL 2016 Shared Task (CoNLL16) find
  39. Top-level The CoNLL 2016 shared task (Xue et al., 2016) provides more abundant annotation than PDTB for shallow discourse parsing. The PDTB section 23 and Wikinews texts following the PDTB annotation guidelines were organized as the test sets. CoNLL16 merges several labels of PDTB. For example, Cont… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-connective-prediction-implicit-discourse-relation/