Elamur Research⌕
☰
home / papers / 2023 / Findings of the Second Shared Task on Multilingual Coreference Resolution

Findings of the Second Shared Task on Multilingual Coreference Resolution

Summary

This paper describes the second round of the multilingual coreference resolution shared task, held with CRAC 2023. Participants built trainable systems that find mentions and cluster them into identity-coreference entities. The data come from CorefUD 1.1, which harmonizes 17 datasets in 12 languages into a common CoNLL-U-based format. Compared with the 2022 edition, the organizers made three main changes:

  • New data: Hungarian KorKor, Norwegian NARC (Bokmål and Nynorsk) and Turkish ITCC were added.
  • Predicted features: gold morpho-syntax in the dev and test inputs was replaced with UDPipe 2 predictions, which makes evaluation more realistic.
  • Head matching: the primary metric now matches mentions by their heads instead of using partial matching, because partial matching had tempted teams to shrink predicted spans to bare heads.

Seven systems and a baseline were compared with the CorefUD scorer, which builds on the Universal Anaphora scorer. CorPipe led on every dataset. The task matters because it pushes coreference research beyond English. It also exposes open problems, including limited typological diversity, uneven annotation quality across datasets, and macro-averaging over corpora of very different sizes.

Key points

  • CorefUD 1.1 contains 17 datasets in 12 languages from 3 language families. Compared with CorefUD 1.0 it adds 4 datasets and 2 languages (Norwegian and Turkish). Train/dev/test splits are roughly 8/1/1.
  • The primary score is the CoNLL F1 with singletons excluded and head-based mention matching. Supplementary scores include MOR and an anaphor-level evaluation of zeros.
  • Participants received input documents with zeros (empty nodes) already reconstructed. The organizers plan to move toward raw-text input in future editions.
  • On the per-dataset CoNLL scores, CorPipe ranked first everywhere. Examples: 83.39 on es_ancora, 82.59 on ca_ancora and 55.63 on tr_itcc. Turkish ITCC was the hardest dataset for all systems, partly because its annotation is incomplete.
  • cs_pcedt is the largest dataset, with about 1.16M words. ParCorFull is among the smallest, with about 11K words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.crac-sharedtask.1

@inproceedings{zabokrtsku2023findings,
  title = {{Findings of the Second Shared Task on Multilingual Coreference Resolution}},
  author = {Žabokrtský, Zdeněk and Konopik, Miloslav and Nedoluzhko, Anna and Novák, Michal and Ogrodniczuk, Maciej and Popel, Martin and Prazak, Ondrej and Sido, Jakub and Zeman, Daniel},
  booktitle = {Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution},
  year = {2023},
  doi = {10.18653/v1/2023.crac-sharedtask.1},
  url = {https://doi.org/10.18653/v1/2023.crac-sharedtask.1}
}

References

48 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563–566. find our card
  3. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT 2005, pages 25–32. Association for Computational Linguistics. ↗ our card
  4. Peter Bourgonje and Manfred Stede. 2020. The Potsdam Commentary Corpus 2.2: Extending Annotations for Shallow Discourse Parsing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1061–1066, Marseille, France. European Language Resources Association. find our card
  5. Petter Mæhlum, Dag Haug, Tollef Jørgensen, Andre Kåsen, Anders Nøklestad, Egil Rønningstad, Per Erik Solberg, Erik Velldal, and Lilja Øvrelid. 2022. NARC–Norwegian anaphora resolution corpus. In Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 48–60, Gyeongju, Korea. Association for Computational Linguistics. find our card
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  7. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  8. Pascal Denis and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, (42):87–96. find
  9. Vladimir Dobrovolskii. 2021. Word-level coreference resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7670–7675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗ our card
  10. Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jiří Mírovský. 2016. Coreference in Prague Czech-English Dependency Treebank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 169–176, Portorož, Slovenia. European Language Resources Association. ↗ our card
  11. Jan Hajič, Eduard Bejček, Jaroslava Hlaváčová, Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpánková. 2020. Prague Dependency Treebank -Consolidated 1.0. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 5208–5218, Marseille, France. European Language Resources Association. ↗
  12. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, and Daniel Zeman. 2022. CorefUD 1.0: Coreference meets Universal Dependencies. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4859–4872, Marseille, France. European Language Resources Association. ↗ our card
  13. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. ↗ our card
  14. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, and Daniel Zeman. 2021. Is one head enough? Mention heads in coreference annotations compared with UD-style heads. In Proceedings of the Sixth International Conference on Dependency Linguistics (Depling, SyntaxFest 2021), pages 101–114, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  15. Frédéric Landragin. 2021. Le corpus Democrat et son exploitation. Présentation. Langages, 224:11–24. ↗
  16. Maciej Ogrodniczuk, Katarzyna Glowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2013. Polish Coreference Corpus. In Human Language Technology. Challenges for Computer Science and Linguistics — 6th Language and Technology Conference (LTC 2013), Revised Selected Papers, volume 9561 of Lecture Notes in Computer Science, pages 215–226. Springer. find
  17. Ekaterina Lapshinova-Koltunski, Christian Hardmeier, and Pauline Krielke. 2018. ParCorFull: a Parallel Corpus Annotated with Full Coreference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association. ↗ our card
  18. Maciej Ogrodniczuk, Katarzyna Głowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2015. Coreference in Polish: Annotation, Resolution and Evaluation. Walter De Gruyter. find
  19. Marta Recasens, Lluís Màrquez, Emili Sapena, M. Antònia Martí, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1–8, Uppsala, Sweden. Association for Computational Linguistics. find our card
  20. Tuğba Pamay and Gülşen Eryiğit. 2018. Turkish Coreference Resolution. In 2018 Innovations in Intelligent Systems and Applications (INISTA), pages 1–7. ↗
  21. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially Annotated Corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  22. Tuğba Pamay Arslan, Kutay Acar, and Gülşen Eryiğit. 2023. Neural End-to-End Coreference Resolution using Morphological Information. In Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution, pages 34–40. find
  23. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207, Brussels, Belgium. Association for Computational Linguistics. find our card
  24. Tuğba Pamay Arslan and Gülşen Eryiğit. 2023. Incorporating Dropped Pronouns into Coreference Resolution: The case for Turkish. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 14–25. find
  25. Milan Straka. 2023. ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution. In Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution, pages 41–51. ↗ our card
  26. Ian Porada and Jackie Chi Kit Cheung. 2023. McGill at CRAC 2023: Multilingual Generalization of Entity-Ranking Coreference Resolution Models. In Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution, pages 52–57. find
  27. Milan Straka and Jana Straková. 2022. ÚFAL CorPipe at CRAC 2022: Effectivity of Multilingual Models for Coreference Resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 28–37. Association for Computational Linguistics. ↗ our card
  28. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card
  29. Mariona Taulé, M. Antònia Martí, and Marta Recasens. 2008. AnCora: Multilevel Annotated Corpora for Catalan and Spanish. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco. European Language Resources Association. ↗
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
  31. Svetlana Toldova, Anna Roytberg, Alina Ladygina, Maria Vasilyeva, Ilya Azerkovich, Matvei Kurzukov, G. Sim, D.V. Gorshkov, A. Ivanova, Anna Nedoluzhko, and Yulia Grishina. 2014. Evaluating Anaphora and Coreference Resolution for Russian. In Komp’juternaja lingvistika i intellektual’nye tehnologii. Po materialam ezhegodnoj Mezhdunarodnoj konferencii Dialog, pages 681–695. find
  32. Ondřej Pražák, Miloslav Konopík, and Jakub Sido. 2021. Multilingual Coreference Resolution with Harmonized Annotations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1119– 1123. ↗ our card
  33. Shubham Toshniwal, Sam Wiseman, Allyson Ettinger, Karen Livescu, and Kevin Gimpel. 2020. Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8519–8526, Online. Association for Computational Linguistics. ↗ our card
  34. Ondřej Pražák and Miloslav Konopík. 2022. End-to-end Multilingual Coreference Resolution with Mention Head Prediction. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 23–27. Association for Computational Linguistics. ↗ our card
  35. Shubham Toshniwal, Patrick Xia, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2021. On generalization in coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 111–120, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗ our card
  36. Marta Recasens and Eduard H. Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. ↗
  37. Don Tuggener. 2014. Coreference resolution evaluation for higher level applications. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 231–235, Gothenburg, Sweden. Association for Computational Linguistics. find our card
  38. Language. In Fifth International Conference on Social Networks Analysis, Management and Security, SNAMS 2018, Valencia, Spain, October 15-18, 2018, pages 243–250. IEEE. find
  39. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, Daniel Zeman, and Yilun Zhu. 2022. Findings of the Shared Task on Multilingual Coreference Resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 1–17, Gyeongju, Korea. Association for Computational Linguistics. ↗
  40. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU Corpus. Natural Language Engineering, 26(1):95–128. ↗
  41. Noémi Vadász. 2022. Building a manually annotated Hungarian coreference corpus: Workflow and tools. In Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 38–47, Gyeongju, Korea. Association for Computational Linguistics. ↗ our card
  42. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. ↗ our card
  43. Veronika Vincze, Klára Hegedűs, Alex Sliz-Nagy, and Richárd Farkas. 2018. SzegedKoref: A Hungarian coreference corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association. ↗ our card
  44. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  45. Juntao Yu, Sopan Khosla, Nafise Sadat Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2022. The Universal Anaphora Scorer. In Proceedings of the Language Resources and Evaluation Conference, pages 4873–4883, Marseille, France. European Language Resources Association. find our card
  46. Juntao Yu, Michal Novák, Abdulrahman Aloraini, Nafise Sadat Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2023. The Universal Anaphora Scorer 2.0. In Proceedings of the 15th International Conference on Computational Semantics (IWCS), Nancy, France. Association for Computational Linguistics. find our card
  47. Amir Zeldes. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  48. Voldemaras Žitkus and Rita Butkienė. 2018. Coreference Annotation Scheme and Corpus for Lithuanian find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abokrtsky-2023-findings-second-shared-task-multilingual/