Elamur Research⌕
☰
home / papers / 2024 / Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation

Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation

Summary

The paper addresses cross-document event coreference resolution (CDCR): deciding whether event mentions in different texts describe the same real-world event. Lexical overlap between triggers can mislead, for example when similar words describe distinct incidents. The authors prompt an open-weight LLM (LLaMA 2-7B-Chat) in a zero-shot, instruction-based way. Given an event pair and its gold label, the model writes step-by-step "inner-monologue" free-text rationales (FTRs) that justify the label.

A smaller Longformer-base student then learns from these rationales in two stages:

  • Rationale-Oriented Event Clustering (ROEC): the student's embedding space is organized around gold event clusters and aligned with the rationale representations.
  • Knowledge distillation: a coreference-specific loss lets the frozen teacher's distribution act as soft supervision.

Rationales are needed only during training, so inference uses the student alone. The approach also skips the costly document-clustering step that many CDCR systems rely on. It reaches state-of-the-art B3 F1 on ECB+ and GVC and sets a first baseline on AIDA Phase 1. This matters because LLM-written explanations can enrich existing corpora and inject reasoning cues without extra human annotation.

Key points

  • Models: the teacher is LLaMA 2-7B-Chat, whose open weights give access to its internal distribution. The student is Longformer-base. Code, weights and FTR sets are released at github.com/csu-signal/llama_cdcr.
  • Corpora: experiments drew on three English CDCR datasets, namely ECB+, AIDA Phase 1, and GVC (short for Gun Violence Corpus). AIDA Phase 1 covers the Russia-Ukraine conflict, is distributed by LDC as LDC2019E77, and has a test set larger than its training set.
  • Candidate pairs: these come from the heuristic filtering of Ahmed et al. (2023) rather than document clustering. The system still exceeds Held et al. (2021) on B3 F1 for ECB+ and GVC.
  • Model refusals: in the zero-shot comparison, LLaMA 2 declined to answer for violence-related content. This affected 167 of 7,314 AIDA Phase 1 pairs (~2.28%) and 80 of 10,355 GVC pairs (~0.77%), and these pairs were excluded. Refusals did not occur during rationale generation.
  • Limitations: the generated rationales sometimes hallucinate, for instance by attributing a fact to both documents when only one states it. Suggested future work includes filtering low-quality FTRs, adding critic models, and generating rationales with GPT-4.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-long.218 · arXiv 2404.03196

@inproceedings{nath2024okay,
  title = {{Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation}},
  author = {Nath, Abhijnan and Avari, Shadi Manafi and Chelle, Avyakta and Krishnaswamy, Nikhil},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.218},
  eprint = {2404.03196},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.218}
}

References

63 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shafiuddin Rehan Ahmed, George Baker, Evi Judge, Michael Regan, Kristin Wright-Bettner, Martha Palmer, and James H. Martin. 2024a. Linear cross-document event coreference resolution with x-amr. find our card
  2. In Proceedings of the Joint Conference of the 15th Language Resources and Evaluation Conference, and, the 30th International Conference on Computational Linguistics, Torino, Italy. European Language Resources Association. find
  3. Shafiuddin Rehan Ahmed, Jon Cai, Martha Palmer, and James H. Martin. 2024b. X-amr annotation tool. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Malta. Association for Computational Linguistics. find
  4. Shafiuddin Rehan Ahmed, Abhijnan Nath, James H. Martin, and Nikhil Krishnaswamy. 2023. 2 ∗ n is better than n 2 : Decomposing event coreference resolution into two tractable problems. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1569–1583, Toronto, Canada. Associatio… find our card
  5. Ben Alderson-Day and Charles Fernyhough. 2015. Inner speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin, 141(5):931. find
  6. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference, volume 1, pages 563–566. Citeseer. find our card
  7. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv e-prints, pages arXiv–2004. find
  8. Barbara L Bershon. 1992. Cooperative problem solving: A link to inner speech. Interaction in cooperative groups. The theoretical anatomy of group learning, pages 36–48. find
  9. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. In International Conference on Learning Representations. find our card
  10. Michael Bugert, Nils Reimers, Shany Barhom, Ido Dagan, and Iryna Gurevych. 2020. Breaking the subtopic barrier in cross-document event coreference resolution. In Text2Story @ ECIR, pages 23–29. find
  11. Michael Bugert, Nils Reimers, and Iryna Gurevych. 2021a. Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora. Computational Linguistics, 47(3):575–614. find our card
  12. Michael Bugert, Nils Reimers, and Iryna Gurevych. 2021b. Generalizing cross-document event coreference resolution across multiple corpora. Computational Linguistics, 47(3):575–614. find our card
  13. Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. 2021. Cdlm: Cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648–2662. find
  14. Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021. Cross-document coreference resolution over predicted mentions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5100–5107. find our card
  15. Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124–2133. find our card
  16. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. find
  17. Andres Cremisini and Mark Finlayson. 2020. New insights into cross-document event coreference: Systematic comparison and a simplified approach. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 1–10, Online. Association for Computational Linguistics… find our card
  18. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4545–4552. find our card
  19. Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–10, Denver, Colorado. Association for Computational Linguisti… find our card
  20. Andrew S Gordon and Jerry R Hobbs. 2017. A formal theory of commonsense psychology: How people think people think. Cambridge University Press. find
  21. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819. find
  22. William Held, Dan Iter, and Dan Jurafsky. 2021. Focus on what matters: Applying discourse coherence theory to cross document coreference. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1406–1417, Online and Punta Cana, Dominican Republic. Association… find our card
  23. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071. find our card
  24. Naoya Inoue, Ekaterina Ovchinnikova, Kentaro Inui, and Jerry R Hobbs. 2012. Coreference resolution with ilp-based weighted abduction. In Proceedings of COLING 2012, pages 1291–1308. find
  25. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. find
  26. Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. arXiv preprint arXiv:1805.10985. find our card
  27. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  28. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. find our card
  29. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  30. Nghia T Le and Alan Ritter. 2023. Are large language models robust zero-shot coreference resolvers? arXiv e-prints, pages arXiv–2305. find
  31. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 489–50… find our card
  32. Nafise Sadat Moosavi, Leo Born, Massimo Poesio, and Michael Strube. 2019. Using automatically extracted minimum spans to disentangle coreference evaluation from boundary detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Flo… find our card
  33. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642. find our card
  34. Shikhar Murty, Pang Wei Koh, and Percy Liang. 2020. ExpBERT: Representation engineering with natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2106–2113, Online. Association for Computational Linguistics. find
  35. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546. find
  36. Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Baker, James H. Martin, Nathaniel Blanchard, and Nikhil Krishnaswamy. 2024. Multimodal cross-document event coreference resolution using linear semantic transfer and mixed-modality ensembles. In Proceedings of the Joint Conference of the 15th… find our card
  37. Abhijnan Nath, Sheikh Mannan, and Nikhil Krishnaswamy. 2023. Axomiyaberta: A phonologically-aware transformer model for assamese. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11629–11646. find
  38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  39. Gabriele Paul. 1993. Approaches to abductive reasoning: an overview. Artificial intelligence review, 7(2):109–152. find
  40. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 968–977, Singapore. Association for Computational Linguistics. find our card
  41. Rajat Raina, Andrew Y Ng, and Christopher D Manning. 2005. Robust textual inference via learning and abductive reasoning. In AAAI, pages 1099–1105. find
  42. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942. find our card
  43. Sahithya Ravi, Chris Tanner, Raymond Ng, and Vered Shwartz. 2023. What happens before and after: Multi-event commonsense in event coreference resolution. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1700–1716. find our card
  44. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, Toronto, Canada. Association for Computational Linguistics. find our card
  45. Charlie Snell, Dan Klein, and Ruiqi Zhong. 2022. Learning by distilling context. find
  46. Aleksandr Sokolov. 2012. Inner speech and thought. Springer Science & Business Media. find
  47. Jiao Sun, Swabha Swayamdipta, Jonathan May, and Xuezhe Ma. 2022. Investigating the benefits of free-form rationales. arXiv preprint arXiv:2206.11083. find
  48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  49. Jennifer Tracey, Ann Bies, Jeremy Getman, Kira Griffitt, and Stephanie Strassel. 2022. A study in contradiction: Data and annotation for aida focusing on informational conflict in russia-ukraine relations. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1831–1838… find
  50. Jingxuan Tu, Kyeongmin Rim, Eben Holderness, Bingyang Ye, and James Pustejovsky. 2023. Dense paraphrasing for textual enrichment. In Proceedings of the 15th International Conference on Computational Semantics, pages 39–49, Nancy, France. Association for Computational Linguistics. find
  51. Marc Vilain, John D Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. find our card
  52. Piek Vossen, Filip Ilievski, Marten Postma, and Rox-ane Segers. 2018. Don’t annotate, but validate: A data-to-text method for capturing event data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  53. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319. find our card
  54. Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Associa… find our card
  55. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  56. Sarah Wiegreffe, Ana Marasović, and Noah A Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266–10284. find
  57. Kazeto Yamamoto, Naoya Inoue, Kentaro Inui, Yuki Arase, and Jun’ichi Tsujii. 2015. Boosting the efficiency of first-order abductive reasoning using pre-estimated relatedness between predicates. International Journal of Machine Learning and Computing, 5(2):114–120. find
  58. Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517– 528. find
  59. Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022. What GPT knows about who is who. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 75– 81, Dublin, Ireland. Association for Computational Linguistics. find
  60. Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2022. Pairwise representation learning for event coreference. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 69–78. find our card
  61. Jin Zhao, Nianwen Xue, and Bonan Min. 2023a. Cross-document event coreference resolution: Instruct humans or instruct GPT? In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 561–574, Singapore. Association for Computational Linguistics. find
  62. Wenting Zhao, Justin T Chiu, Claire Cardie, and Alexander M Rush. 2023b. Abductive commonsense reasoning exploiting mutually exclusive explanations. arXiv preprint arXiv:2305.14618. find
  63. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097–1100. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nath-2024-okay-let-s-do-this/