Elamur Research⌕
☰
home / papers / 2022 / ConnPrompt: Connective-cloze Prompt Learning for Implicit Discourse Relation Recognition

ConnPrompt: Connective-cloze Prompt Learning for Implicit Discourse Relation Recognition

Summary

Implicit Discourse Relation Recognition (IDRR) assigns a relation sense to two adjacent text segments (arguments) that have no explicit connective linking them. Most prior systems add a task-specific neural network on top of a pre-trained language model (PLM) and fine-tune it. This approach needs many labelled examples, and its training objective differs from the masked-language-model objective used in pre-training. The authors propose ConnPrompt, which they present as the first application of prompt learning to IDRR. The PLM fills a masked slot with a connective word, and a fixed verbalizer maps that word to a top-level PDTB sense. They build three cloze templates and a 16-word answer space drawn from PDTB connectives, then combine the templates' outputs by voting. Experiments on PDTB 3.0 use BERT, RoBERTa, ERNIE and DeBERTa. The authors report that ConnPrompt significantly beats state-of-the-art models with the full training set and stays competitive with reduced training data. This suggests that the connective knowledge already present in masked PLMs can stand in for heavy task-specific architectures.

Key points

  • Templates. There are two template styles:
    • Insert-cloze Prompt (ICP) places [MASK] between the two concatenated arguments.
    • Prefix-cloze Prompt (PCP) separates the arguments with [SEP] and puts [MASK] in front of either Arg1 or Arg2.
    • Together these give three templates.
  • Verbalizer. Sixteen single-word, unambiguous, frequent connectives were selected from about 200 in PDTB. Each word maps to one of four classes: Comparison, Contingency, Expansion or Temporal. During training, an instance whose gold connective falls outside this set is labelled with the most frequent answer word sharing its subtype-level sense.
  • Ensembling. The final label is decided by majority vote across the three templates. If all three disagree, the prediction of the template with the best validation F1 is used.
  • Data and training. The corpus is PDTB 3.0, split into sections 2–20 for training, 0–1 for development and 21–22 for testing. The PLM is tuned with cross-entropy loss and the AdamW optimizer; the verbalizer itself adds no trainable parameters.
  • Few-shot experiments. Training sets were created by down-sampling. RoBERTa was the strongest backbone in this setting. ConnPrompt was compared with the BERT-based IPAL and PLR models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

21 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xiang2022connprompt,
  title = {{ConnPrompt: Connective-cloze Prompt Learning for Implicit Discourse Relation Recognition}},
  author = {Xiang, Wei and Wang, Zhenglin and Dai, Lu and Wang, Bang},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  url = {https://aclanthology.org/2022.coling-1.75/}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571– 583. find our card
  2. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1726–1735. find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  4. Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. 2021. Prompt-learning for fine-grained entity typing. arXiv preprint, arXiv:2108.10604:1–12. find
  5. Fengyu Guo, Ruifang He, Jianwu Dang, and Jian Wang. 2020. Working memory-driven neural networks with a novel knowledge enhancement paradigm for http://www.mindspore.cn/implicit discourse relation recognition. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 7822–… ↗
  6. Francisco Guzmán, Shafiq Joty, Lluís Màrquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 687–698. find
  7. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, pages 1–23. find our card
  8. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  9. Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recognizing implicit discourse relation on joint modeling of classification and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page 2418–2431. find our card
  10. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299– 1308. find our card
  11. Wenqiang Lei, Xuancong Wang, Meichun Liu, Ilija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: a simple word interaction model for implicit discourse relation recognition. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 4026–4032. find
  12. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059. find
  13. Xiao Li, Yu Hong, Huibin Ruan, and Zhen Huang. 2020. Using a penalty-based loss re-estimation method to improve implicit discourse relation classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1513–1518. find
  14. Maria Liakata, Simon Dobnik, Shyamasree Saha, Colin Batchelor, and Dietrich Rebholz Schuhmann. 2013. A discourse-driven content model for summarising scientific articles evaluated in a complex question answering task. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pro… find
  15. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint, arXiv:2107.13586:1–46. find our card
  16. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3830–3836. find
  17. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224–1233. find our card
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint, arXiv:1907.11692:1–13. find
  19. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, pages 1– 18. find
  20. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K. Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 87–90. find our card
  21. Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, and Min Zhang. 2020. Interactively-propagative attention learning for implicit discourse relation recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3168–3178. find our card
  22. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 281–291. find our card
  23. Ronald Seoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021. Open aspect target sentiment classification with natural language prompts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6311–6322. find
  24. Wei Shi and Vera Demberg. 2019. Learning to explic-itate connectives with seq2seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, page 188–199. find
  25. Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: enhanced representation through knowledge integration. arXiv preprint, arXiv:1904.09223:1–8. find our card
  26. Chang Wang and Bang Wang. 2020. An end-to-end topic-enhanced self-attention network for social emotion classification. In Proceedings of The Web Conference 2020, pages 2210–2219. find
  27. Chengyu Wang, Jianing Wang, Minghui Qiu, Jun Huang, and Ming Gao. 2021. Transprompt: Towards an automatic transferable prompting framework for few-shot text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2792–2802. find
  28. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania, pages 1–81. find
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  30. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, pages 11486–11494. find
  31. Wei Xiang and Bang Wang. 2019. A survey of event extraction from text. IEEE Access, 7:173111–173137. find
  32. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. find our card
  33. Yingxue Zhang, Fandong Meng, Li Peng, Jian Ping, and Jie Zhou. 2021. Context tracking network: Graph-based context modeling for implicit discourse relation recognition. In Proceedings of the 2021 Conference of the North American Chapter of the Association Computational Linguistics, pages 1592–1599. find our card
  34. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, page 207–212. find our card

↑ back to top