Elamur Research⌕
☰
home / papers / 2024 / A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution

A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution

Summary

Cross-document event coreference resolution (ECR) groups event mentions from different documents that describe the same real-world event. The authors find that the leading pairwise system (Held et al., 2021, a RoBERTa-large cross-encoder) depends too heavily on whether the two event triggers look alike lexically. Trigger similarity is skewed toward coreferent pairs in training data, so this shortcut gets in the way of reading the actual arguments: participants, time, place and action. They frame the model's decision with a Structural Causal Model, which treats trigger matching as a spurious path and argument coreference as the true rationale. They then propose LLM-RCDA, a counterfactual augmentation scheme with an LLM in the loop. It has two parts:

  • Trigger intervention substitutes lexically different synonyms for triggers.
  • Context intervention minimally rewrites the rationale arguments to flip the label, keeping the rest of the sentence intact.

The model architecture is unchanged; only the training data grows. The enhanced system reaches state-of-the-art results on ECB+, FCC and GVC and generalizes better out of domain. This makes it a cheap, model-agnostic way to debias pairwise classifiers.

Key points

  • In-domain results: gains of 1.8–2.3 CoNLL F1 over baselines on ECB+, Football Coreference Corpus (FCC) and Gun Violence Corpus (GVC).
  • Out-of-domain test (train on ECB+, test on FCC): CoNLL F1 rises from 42.5 to 49.9, a gain of about 7.4. The introduction reports this gain as 7.2.
  • LLM generator and zero-shot comparison: the counterfactual data are generated with GPT-3.5-turbo. The authors claim to be the first to benchmark zero-shot Claude-2 and GPT-4 on cross-document ECR, and their fine-tuned system clearly outperforms both on ECB+.
  • Error analysis: the authors sampled 100 baseline errors that the enhanced system fixes. Half of the false positives came from ignoring argument counterparts, such as different opposing teams in otherwise similar sentences. Among false negatives, 32% needed deeper contextual understanding and 24% involved missed argument counterparts.
  • Code and limitations: code is available at github.com/Danield21/Rationale4CDECR. A stated limitation is the reliance on a closed LLM; the authors plan to try open-source models such as LLaMA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-long.63 · arXiv 2404.01921

@inproceedings{ding2024rationale,
  title = {{A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution}},
  author = {Ding, Bowen and Min, Qingkai and Ma, Shengkun and Li, Yingjie and Yang, Linyi and Zhang, Yue},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.63},
  eprint = {2404.01921},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.63}
}

References

54 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shafiuddin Rehan Ahmed, Abhijnan Nath, James H. Martin, and Nikhil Krishnaswamy. 2023. 2 ∗ n is better than n 2 : Decomposing event coreference resolution into two tractable problems. find our card
  2. James Allan, Jaime G. Carbonell, George R. Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic detection and tracking pilot study final report. find
  3. Rohan Anil, Andrew M. Dai, and et al. 2023. Palm 2 technical report. find our card
  4. Anthropic. 2023. Ai research and products that put safety at the frontier. find
  5. Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79–85, Montreal, Quebec, Canada. Associa… find
  6. Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. find our card
  7. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala, Sweden. Association for Computational Linguistics. find our card
  8. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. find
  9. Michael Bugert, Nils Reimers, and Iryna Gurevych. 2021. Generalizing cross-document event coreference resolution across multiple corpora. Computational Linguistics, 47(3):575–614. find our card
  10. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4545–4552, Reykjavik, Iceland. European Language Resources Associa… find our card
  11. Agata Cybulska and Piek Vossen. 2015. " bag of events" approach to event coreference resolution. supervised classification of event templates. Int. J. Comput. Linguistics Appl., 6(2):11–27. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  13. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne,… find our card
  14. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. find
  15. Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. find our card
  16. Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, and Ido Dagan. 2021. Cdlm: Cross-document language modeling. find
  17. William Held, Dan Iter, and Dan Jurafsky. 2021. Focus on what matters: Applying discourse coherence theory to cross document coreference. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1406–1417, Online and Punta Cana, Dominican Republic. Association… find our card
  18. Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2020. Streamlining cross-document coreference resolution: Evaluation and modeling. find
  19. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. 1997. Event coreference for information extraction. In Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. find our card
  20. Xinyu Chen, Sheng Xu, Peifeng Li, and Qiaoming Zhu. 2023a. Cross-document event coreference resolution on discourse structure. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4833–4843, Singapore. Association for Computational Linguistics. find our card
  21. Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. find
  22. Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023b. DISCO: Distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5514–5528, Toronto, Canada.… find
  23. Mark T. Keane and Barry Smyth. 2020. Good counterfactuals and where to find them: A case-based technique for generating counterfactuals for explainable ai (xai). find
  24. Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1–10, New Orleans, Louisiana. Asso… find our card
  25. Nghia T. Le and Alan Ritter. 2023. Are large language models robust zero-shot coreference resolvers? find
  26. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  27. Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2023. Large language models as counterfactual generator: Strengths and weaknesses. find
  28. Qi Liu, Matt Kusner, and Phil Blunsom. 2021. Counterfactual data augmentation for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 187–197, Online. Association for Comp… find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  30. Marcel Robeer, Floris Bex, and Ad Feelders. 2021. Generating realistic natural language counterfactuals. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3611–3625, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  31. Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. 2020. Robustness to spurious correlations via human annotations. find
  32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  33. Mycal Tucker, Peng Qian, and Roger Levy. 2021. What if this modified that? syntactic interventions via counterfactual embeddings. find
  34. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. find our card
  35. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, page 25–32, USA. Association for Computational Linguistics. find our card
  36. Piek Vossen, Filip Ilievski, Marten Postma, and Rox-ane Segers. 2018. Don’t annotate, but validate: a data-to-text method for capturing event data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources A… find
  37. Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Dip-tikalyan Saha. 2021. Generate your counterfactuals: Towards controlled counterfactual generation for text. find
  38. Zhao Wang and Aron Culotta. 2020. Robustness to spurious correlations in text classification via automatically generated counterfactuals. find
  39. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Associ… find our card
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. find our card
  41. OpenAI. 2023. Creating safe agi that benefits all of humanity. find
  42. Palantir. 2023. Ai-powered operations, for every decision. find
  43. Judea Pearl. 2009. Causality: Models, Reasoning and Inference, 2nd edition. Cambridge University Press, USA. find
  44. Quintin Pope and Xiaoli Z. Fern. 2021. Text counterfactuals via latent optimization and shapley-guided search. find
  45. Sahithya Ravi, Chris Tanner, Raymond Ng, and Vered Shwartz. 2023. What happens before and after: Multi-event commonsense in event coreference resolution. find our card
  46. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S. Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. find
  47. Linyi Yang, Jiazheng Li, Pádraig Cunningham, Yue Zhang, Barry Smyth, and Ruihai Dong. 2022a. Exploring the efficacy of automatically generated counterfactuals for sentiment analysis. find
  48. Linyi Yang, Lifan Yuan, Leyang Cui, Wenyang Gao, and Yue Zhang. 2022b. Factmix: Using a few labeled in-domain examples to generalize to cross-domain named entity recognition. find
  49. Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023a. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. find
  50. Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, and Yue Zhang. 2023b. Supervised knowledge makes large language models better in-context learners. find
  51. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… find our card
  52. Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2022. Pairwise representation learning for event coreference. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 69–78, Seattle, Washington. Association for Computational Linguistics. find our card
  53. Yutao Zeng, Xiaolong Jin, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2020. Event coreference resolution with their paraphrases and argument-aware embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3084–3094, Barcelona, Spain (Online). International C… find our card
  54. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International… find our card

↑ back to top