Elamur Research⌕
☰
home / papers / 2023 / Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition

Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition

Summary

The paper addresses multi-level implicit discourse relation recognition (MIDRR): predicting a hierarchy of discourse senses between two arguments that lack an explicit connective. The levels are a top-level class, a second-level sense and an annotator-inserted connective. The authors argue that fully fine-tuning PLMs on scarce IDRR data distorts the pre-trained feature space, and that learning hierarchical semantics needs even more data. Their framework, PEMI, keeps the masked language model frozen. Learnable soft prompt tokens around the arguments adapt the input to the pre-trained space, an approach inspired by WARP. A new verbalizer method, hierarchical label refining (HLR), learns embeddings only for bottom-level (connective) labels. Higher-level label representations are built from these bottom up through learned weighted aggregation, and all levels are trained jointly. On PDTB 2.0 and PDTB 3.0, PEMI performs comparably to recent state-of-the-art systems while training roughly 0.1% of their parameters. It uses no extra data or external knowledge. This makes it an attractive option for low-resource discourse tasks.

Key points

  • Trainable parameters total about 100K, roughly 0.1% of the fine-tuned baselines. The transformer encoder stays frozen, and only soft prompts and verbalizer components are updated.
  • Ablation on PDTB 2.0 compares a fine-tuned RoBERTa MLM baseline, baseline+HLR, and full PEMI. HLR helps most at level 2. The full PEPT+HLR combination scores best.
  • Template design matters:
    • It works better to put the <mask> token between the two arguments.
    • Spreading prompt tokens around the arguments beats compact placement, especially with more tokens near Arg1.
    • 20 prompt tokens was chosen as the overall best setting, although connective accuracy kept rising with more tokens.
  • The HLR weights learned by the model are roughly inversely proportional to class frequency, so minor senses get more attention. Visualizations show connectives clustering by sense.
  • The authors acknowledge several limitations:
    • HLR gives little gain at the top and bottom levels.
    • Few baselines exist for comparison on PDTB 3.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.357 · arXiv 2402.15080

@inproceedings{zhao2023infusing,
  title = {{Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition}},
  author = {Zhao, Haodong and He, Ruifang and Xiao, Mengnan and Xu, Jing},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.357},
  eprint = {2402.15080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.357}
}

References

51 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL. find our card
  2. Hongxiao Bai and Zhao Hai. 2018. Deep enhanced representation for implicit discourse relation recognition. COLING. find our card
  3. Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware label semantics matching network for hierarchical text classification. ACL. find
  4. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. ACL. find our card
  5. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. 2019. Class-balanced loss based on effective number of samples. CVPR. find
  6. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. NAACL. find our card
  7. Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. EMNLP (Short). find our card
  8. Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. 2022. Delta tuning: A comprehensive study of parameter ef… find
  9. Zujun Dou, Yu Hong, Yu Sun, and Guodong Zhou. 2021. Cvae-based re-anchoring for implicit discourse relation classification. EMNLP Findings. find our card
  10. Xiachong Feng, Xiaocheng Feng, Bing Qin, and Xinwei Geng. 2021. Dialogue discourse-aware graph model and data augmentation for meeting summarization. IJCAI. find
  11. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. Ppt: Pre-trained prompt tuning for few-shot learning. ACL. find our card
  12. Zhenwen Li, Wenhao Wu, and Sujian Li. 2020b. Composing elementary discourse units in abstractive summarization. ACL. find
  13. Fengyu Guo, Ruifang He, J. Dang, and Jian Wang. 2020. Working memory-driven neural networks with a novel knowledge enhancement paradigm for implicit discourse relation recognition. AAAI. find
  14. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2022. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR). find our card
  15. Fengyu Guo, Ruifang He, Di Jin, J. Dang, Longbiao Wang, and Xiangang Li. 2018. Implicit discourse relation recognition using neural tensor network with interactive attention and sparse learning. COLING. find our card
  16. Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv. find our card
  17. Karen Hambardzumyan, H. Khachatrian, and Jonathan May. 2021. Warp: Word-level adversarial reprogramming. ACL. find
  18. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. ArXiv. find our card
  19. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification. ArXiv. find
  20. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. IJCAI. find
  21. Ruifang He, Jian Wang, Fengyu Guo, and Yugui Han. 2020. Transs-driven joint learning architecture for implicit discourse relation recognition. ACL. find our card
  22. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. TACL. find our card
  23. Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recognizing implicit discourse relation on joint modeling of classification and generation. EMNLP. find our card
  24. Najoung Kim, Song Feng, R. Chulaka Gunasekara, and L. Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. ACL. find our card
  25. Yudai Kishimoto, Yugo Murawaki, and S. Kurohashi. 2018. A knowledge-augmented neural network model for implicit discourse relation classification. COLING. find
  26. Yudai Kishimoto, Yugo Murawaki, and S. Kurohashi. 2020. Adapting bert to implicit discourse relation classification with a focus on discourse connectives. LREC. find our card
  27. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. EMNLP. find our card
  28. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. EMNLP. find
  29. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. EMNLP. find our card
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Facebook AI. find
  31. L. T. Nguyen, Ngo Van Linh, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. ACL. find our card
  32. Emily Pitler, Annie Louis, and A. Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. ACL. find our card
  33. R. Prasad, N. Dinesh, Alan Lee, Eleni Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008. The penn discourse treebank 2.0. LREC. find our card
  34. Lianhui Qin, Zhisong Zhang, and Zhao Hai. 2016. A stacking gated neural architecture for implicit discourse relation classification. EMNLP. find our card
  35. Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, and Min Zhang. 2020. Interactively-propagative attention learning for implicit discourse relation recognition. COLING. find our card
  36. Wei Shi and V. Demberg. 2021. Entity enhancement for implicit discourse relation classification in the biomedical domain. ACL. find
  37. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL. find
  38. Wei Shi, Frances Yung, and V. Demberg. 2018. Acquiring annotated data with cross-lingual explicitation for implicit discourse relation classification. DISRPT. find our card
  39. Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020a. Dice loss for data-imbalanced nlp tasks. ACL. find
  40. Shivashankar Subramanian, Afshin Rahimi, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Fairness-aware class imbalanced learning. EMNLP. find
  41. Jialong Tang, Hongyu Lin, M. Liao, Yaojie Lu, Xian-pei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. ACL. find
  42. Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. ACL. find
  43. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-aware sequence generation model for multi-level implicit discourse relation recognition. AAAI. find
  44. Changxing Wu, Chaowen Hu, Ruochen Li, Hongyu Lin, and Jinsong Su. 2020. Hierarchical multi-task learning with crf for implicit discourse relation recognition. Knowledge Base System. find
  45. Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022a. Encoding and fusing semantic connection and linguistic evidence for implicit discourse relation recognition. ACL Findings. find our card
  46. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022b. Connprompt: Connective-cloze prompt learning for implicit discourse relation recognition. COL-ING. find our card
  47. Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, and Guodong Zhou. 2018. Using active learning to expand training data for implicit discourse relation recognition. EMNLP. find our card
  48. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. EMNLP. find our card
  49. Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Differentiable prompt makes pre-trained language models better few-shot learners. NeurIPS, abs/2108.13161. — [arXiv:2108.13161](https://arxiv.org/abs/2108.13161) find
  50. Yingxue Zhang, Fandong Meng, Peng Li, Ping Jian, and Jie Zhou. 2021. Context tracking network: Graph-based context modeling for implicit discourse relation recognition. NAACL. find our card
  51. Hao Zhou, Man Lan, Yuanbin Wu, YueFeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. EMNLP Findings, abs/2210.07032. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2023-infusing-hierarchical-guidance-into-prompt/