Elamur Research⌕
☰
home / papers / 2024 / Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

Summary

The paper tackles cross-document event coreference resolution (ECR): deciding whether event mentions in different documents describe the same real-world occurrence, even when the wording differs or is ambiguous. The authors argue that accompanying images can resolve such cases, but standard benchmarks contain almost no visual data. To address this, they enrich ECB+ with images recovered from the original news articles and archived copies, and fill the remaining gaps with Stable Diffusion images generated for individual mentions. They then compare three ways of combining modalities. The first is a conventional fused model that is fine-tuned. The second is Lin-Sem, a cheap bidirectional mapping between vision and text Transformer embeddings fitted with ridge regression and requiring no fine-tuning. The third is an ensemble that sends easy mention pairs to a text-only Longformer and hard pairs to multimodal models. The best ensembles reach 91.9 CoNLL F1 on ECB+, which the authors frame as an upper bound under oracle pruning. The paper also reports the first results on the multimodal AIDA Phase 1 corpus, showing that images help on difficult pairs.

Key points

  • ECB+ image augmentation: web scraping and the Internet Archive yielded 543 images, but only 235 of 982 documents had any image. Diffusion-generated pictures supplied images for every event mention.
  • Lin-Sem: fits ridge regression (λ=1) in both directions, text→image and image→text. The text encoder is Longformer, and the vision encoders are ViT, BEiT, SWIN and CLIP.
  • Ensemble design: mention pairs are split by semantic and discourse difficulty into easy, hard-positive and hard-negative pairs. Easy pairs always go to the text model. The models for hard pairs were chosen by grid search.
  • AIDA Phase 1 (LDC2019E77): this corpus covers Russia–Ukraine events, and only its English documents were used. The best ensemble scores 64.6 CoNLL F1, compared with 61.4 for the text-only Longformer baseline.
  • Pruning assumption: candidate pairs are pruned using the "oracle" lemma-based method of Ahmed et al. (2023). Code is available at github.com/csu-signal/multimodal-coreference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Coreference resolution

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2404.08949

@inproceedings{nath2024multimodal,
  title = {{Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles}},
  author = {Nath, Abhijnan and Jamil, Huma and Ahmed, Shafiuddin Rehan and Baker, George Arthur and Ghosh, Rahul and Martin, James H. and Blanchard, Nathaniel and Krishnaswamy, Nikhil},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/2vi8vepe3v7j},
  eprint = {2404.08949},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/2vi8vepe3v7j}
}

References

51 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shafiuddin Rehan Ahmed, Abhijnan Nath, James H. Martin, and Nikhil Krishnaswamy. 2023. 2 ∗ n is better than n 2 : Decomposing event coreference resolution into two tractable problems. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1569–1583, Toronto, Canada. Association for Computational Linguistics. ↗ our card
  2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference, volume 1, pages 563–566. Citeseer. find our card
  3. Amit Bagga and Breck Baldwin. 1999. Cross-document event coreference: Annotations, experiments, and observations. In Coreference and Its Applications. find our card
  4. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations. ↗
  5. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv e-prints, pages arXiv–2004. find
  6. Michael Bugert, Nils Reimers, Shany Barhom, Ido Dagan, and Iryna Gurevych. 2020. Breaking the subtopic barrier in cross-document event coreference resolution. In Text2story@ ecir, pages 23–29. find
  7. Michael Bugert, Nils Reimers, and Iryna Gurevych. 2021. Generalizing cross-document event coreference resolution across multiple corpora. Computational Linguistics, 47(3):575–614. ↗ our card
  8. Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648–2662, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  9. Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021. Cross-document coreference resolution over predicted mentions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5100–5107, Online. Association for Computational Linguistics. find our card
  10. Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih-Fu Chang. 2021. Joint multimedia event extraction from video and article. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 74–88. find
  11. Lin Chen, Anruo Wang, and Barbara Di Eugenio. 2011. Improving pronominal and deictic coreference resolution with multi-modal features. In Proceedings of the SIGDIAL 2011 Conference, pages 307–311. find
  12. Zheng Chen and Heng Ji. 2009. Graph-based event coreference resolution. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4), pages 54–57, Suntec, Singapore. Association for Computational Linguistics. find our card
  13. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4545–4552. find our card
  14. Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–10, Denver, Colorado. Association for Computational Linguistics. find our card
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina N Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. find
  16. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. ↗
  17. Jacob Eisenstein and Randall Davis. 2006. Gesture improves coreference resolution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 37–40, New York City, USA. Association for Computational Linguistics. find our card
  18. Andrew Finch, Young-Sook Hwang, and Eiichiro Sumita. 2005. Using machine translation evaluation techniques to determine sentence-level semantic equivalence. In Proceedings of the third international workshop on paraphrasing (IWP2005). find
  19. Sadaf Ghaffari and Nikhil Krishnaswamy. 2023. Grounding and distinguishing conceptual vocabulary through similarity learning in embodied simulations. In Proceedings of the 15th International Conference on Computational Semantics. ↗
  20. Danfeng Guo, Arpit Gupta, Sanchit Agarwal, Jiun-Yu Kao, Shuyang Gao, Arijit Biswas, Chien-Wei Lin, Tagyoung Chung, and Mohit Bansal. 2022. Gravl-bert: Graphical visual-linguistic representations for multimodal coreference resolution. In Proceedings of the 29th International Conference on Computational Linguistics, pages 285– 297. find our card
  21. William Held, Dan Iter, and Dan Jurafsky. 2021. Focus on what matters: Applying discourse coherence theory to cross document coreference. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1406–1417, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  22. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations. find
  23. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. 1997. Event coreference for information extraction. In Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. find our card
  24. Huma Jamil, Yajing Liu, Nathaniel Blanchard, Michael Kirby, and Chris Peterson. 2023. Leveraging linear mapping for model-agnostic adversarial defense. Frontiers in Computer Science, 5. ↗
  25. Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1– 10. find our card
  26. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  27. Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612–5623. find
  28. Karel Lenc and Andrea Vedaldi. 2015. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991–999. find
  29. Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2557–2568. find
  30. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022. find our card
  31. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32. find our card
  32. David McNeely-White, J Ross Beveridge, and Bruce A Draper. 2020. Inception and resnet features are (almost) equivalent. Cognitive Systems Research, 59:312–318. ↗
  33. David McNeely-White, Ben Sattelberg, Nathaniel Blanchard, and Ross Beveridge. 2022. Canonical face embeddings. IEEE Transactions on Biometrics, Behavior, and Identity Science. ↗
  34. Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly mapping from image to text space. In International Conference on Learning Representations. ↗
  35. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  36. Nafise Sadat Moosavi, Leo Born, Massimo Poesio, and Michael Strube. 2019. Using automatically extracted minimum spans to disentangle coreference evaluation from boundary detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4168–4178, Florence, Italy. Association for Computational Linguistics. ↗ our card
  37. Abhijnan Nath, Rahul Ghosh, and Nikhil Krishnaswamy. 2022. Phonetic, semantic, and articulatory features in Assamese-Bengali cognate detection. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 41–53, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  38. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR. ↗ our card
  39. Sahithya Ravi, Chris Tanner, Raymond Ng, and Vered Shwartz. 2023. What happens before and after: Multi-event commonsense in event coreference resolution. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1700–1716. find our card
  40. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695. ↗
  41. Roger C Schank and Robert P Abelson. 1975. Scripts, plans, and knowledge. In IJCAI, volume 75, pages 151–157. find
  42. Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111. ↗ our card
  43. Meihan Tong, Shuai Wang, Yixin Cao, Bin Xu, Juanzi Li, Lei Hou, and Tat-Seng Chua. 2020. Image enhanced event detection in news articles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9040– 9047. find
  44. Jennifer Tracey, Ann Bies, Jeremy Getman, Kira Griffitt, and Stephanie Strassel. 2022. A study in contradiction: Data and annotation for aida focusing on informational conflict in russia-ukraine relations. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1831–1838. find
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find
  46. Marc Vilain, John D Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. find our card
  47. Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. 2013. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision, pages 1–8. find
  48. Piek Vossen, Filip Ilievski, Marten Postma, and Roxane Segers. 2018. Don’t annotate, but validate: A data-to-text method for capturing event data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  49. Zhibiao Wu and Martha Palmer. 1994. Verb semantics and lexical selection. In 32nd Annual Meeting of the Association for Computational Linguistics, pages 133–138. ↗
  50. Nishant Yadav, Nicholas Monath, Rico Angell, and Andrew McCallum. 2021. Event and entity coreference using trees to encode uncertainty in joint decisions. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 100–110. find
  51. Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2022. Pairwise representation learning for event coreference. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 69–78. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nath-2024-multimodal-cross-document-event-coreference/