Elamur Research⌕
☰
home / papers / 2024 / An Empirical Study of Synthetic Data Generation for Implicit Discourse Relation Recognition

An Empirical Study of Synthetic Data Generation for Implicit Discourse Relation Recognition

Summary

Implicit Discourse Relation Recognition (IDRR) asks a model to identify the semantic link between two adjacent text spans that have no explicit connective. Some relations are rare in the training data and frequently misclassified. Omura, Cheng and Kurohashi address this by having a large language model create additional training examples for those weak classes.

Their error-driven pipeline first reads a confusion matrix from a fine-tuned RoBERTa classifier and selects relation pairs the model confuses, ranked by false negative rate. It then generates argument pairs aimed at those confusions through two-stage prompting. Each stage requires the LLM to learn only one relation from few-shot examples, which is easier than separating all 14 at once. The synthetic examples, several times larger in number than the original data for the targeted classes, are added to training on PDTB-3.

The authors report state-of-the-art macro-F1 for second-level classification with no loss in micro-F1, and the clearest gains come on infrequent relations. This matters because expert annotation of discourse relations is costly.

Key points

  • Setup: PDTB-3 with 14-way second-level classification (only labels with more than 100 examples, following Kim et al., 2020). It uses the Ji and Eisenstein (2015) split: sections 2–20 for training, 0–1 for development, and 21–22 for test.
  • Few-shot GPT-3.5 and GPT-4 perform far below fine-tuned RoBERTa. On top-level relations (derived from second-level predictions), micro/macro-F1 is about 41.5/36.5 for GPT-3.5 and 42.3/42.2 for GPT-4, compared with 78.0/74.0 for vanilla RoBERTa LARGE.
  • In the preliminary analysis, RoBERTa tends to mistake rare relations for frequent ones rather than for sibling relations under the same top-level class. This questions hierarchy-based hard-negative approaches.
  • Synthetic data is generated for only a few error-prone relations and released in unfiltered and filtered versions. For example, one targeted class receives about 2,500 unfiltered (about 1,300 filtered) examples, compared with 435 or fewer gold examples.
  • Synthetic data and code are available at github.com/ku-nlp/sdg4idrr.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{omura2024empirical,
  title = {{An Empirical Study of Synthetic Data Generation for Implicit Discourse Relation Recognition}},
  author = {Omura, Kazumasa and Cheng, Fei and Kurohashi, Sadao},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.96/}
}

References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prajjwal Bhargava and Vincent Ng. 2022. Dis-coSense: Commonsense reasoning with discourse connectives. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10295–10310, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Wong, and Simon See. 2023. DiscoPrompt: Path prediction prompt tuning for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 35–57, Toronto, Canada. Association for… find our card
  4. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xi-aoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. AugGPT: Leveraging Chat-GPT for Text Data Augmentation. find
  5. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022a. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computa… find our card
  6. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistic… find our card
  7. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–11… find our card
  8. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665–673. find
  9. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  10. Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recognizing implicit discourse relation on joint modeling of classification and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2418–2431, Onl… find our card
  11. Yuxin Jiang, Linhan Zhang, and Wei Wang. 2023. Global and local hierarchy-aware contrastive framework for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8048–8064, Toronto, Canada. Association for Computational Linguistics. find our card
  12. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404–5414, Online. Association for Computational Linguis… find our card
  13. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1152–1158, Marseille, France. European Language Resources As… find our card
  14. Wei Liu and Michael Strube. 2023. Annotation-inspired implicit discourse relation classification with auxiliary discourse connective generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15696–15712, Toronto, Canada. Ass… find our card
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  16. Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10704–10716, Abu Dhabi, United Arab Emirates. Associa… find our card
  17. Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. 2023. Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning. find
  18. Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2021. Long-tail learning via logit adjustment. In International Conference on Learning Representations. find
  19. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab… find our card
  20. Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510, Florence, Italy. Association for Computational Linguistics. find our card
  21. Kazumasa Omura and Sadao Kurohashi. 2022. Improving commonsense contingent reasoning by pseudo-data and its application to the related tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 812–823, Gyeongju, Republic of Korea. International Committee on Compu… find
  22. OpenAI. 2023. GPT-4 Technical Report. find our card
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  24. Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5811–5826, Online. Association for Computational Ling… find
  25. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006– 1017, Vanc… find our card
  26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  27. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7… find our card
  28. Jun Saito, Yugo Murawaki, and Sadao Kurohashi. 2019. Minimally supervised learning of affective events using discourse relations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP… find
  29. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6943–6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  30. Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xianpei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International J… find
  31. Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A Label Dependence-Aware Sequence Generation Model for Multi-Level Implicit Discourse Relation Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11486– 11494. find
  32. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911, Gyeongju, Republic of Korea. International Committee on Co… find our card
  33. Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, and Guodong Zhou. 2018. Using active learning to expand training data for implicit discourse relation recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 725–731, Brussels, Belgium. Assoc… find our card
  34. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1008–1025,… find
  35. Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3848–3858, Abu Dhabi, United Arab Emirates. Associati… find our card
  36. Language Resource References Rashmi Prasad and Alan Lee and Nikhil Dinesh and Eleni Miltsakaki and Geraud Campion and Aravind Joshi and Bonnie Webber. 2008. Penn Discourse Treebank Version 2.0. distributed via Linguistic Data Consortium, ISLRN 488-589-036-315-2. find
  37. Rashmi Prasad and Bonnie Webber and Alan Lee and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. distributed via Linguistic Data Consortium, ISLRN 977-491-842-427-0. find

↑ back to top