Elamur Research⌕
☰
home / papers / 2023 / ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution

ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution

Summary

The paper describes CorPipe, the system that took first place in the multilingual coreference resolution competition held at CRAC 2023. The task covers 17 corpora in 12 languages from CorefUD 1.1. CorPipe builds on the author's CRAC 2022 winner. It first detects mentions and then links them to antecedents. Both steps share one pretrained encoder and are trained jointly on all corpora.

The main changes are:

  • Larger encoders. CorPipe uses mT5 large/xl. Their relative positional embeddings allow inputs longer than 512 subwords. Training uses shorter inputs, while prediction uses a much wider context.
  • New mention decoding. The CRF tagger was replaced by independent per-token classification. A constrained dynamic-programming (Viterbi-style) decoder then guarantees that the stack instructions are well balanced. Because each token's prediction is now a separate probability distribution, several models can be ensembled by averaging them.

CorPipe beat the next participant by 4.5 percentage points (CoNLL score). This shows that a wider document context and model ensembling clearly help multilingual coreference. The source code is public.

Key points

  • Official test results: CorPipe scored 74.90 CoNLL, ahead of the runner-up at 70.41 and the Pražák et al. (2021) baseline at 56.96. It ranked first on every one of the 17 corpora.
  • The tags no longer depend on stack depth, which cuts the tag inventory from 207 to 54 in the training data. Development experiments show that a stack depth of 3 is enough.
  • Ensembling five runs adds about 1.0 point for mT5-large and 0.8 for mT5-xl at a context of 512. At a context of 2560, the gains are 1.3 and 1.6 points.
  • The primary submission samples the corpora with logarithmic mix ratios and adds corpus-id tokens. Ablations show that both choices change results only slightly.
  • In a zero-shot setting, where the target language is left out of training, scores fall by 13.2 points. Training with mT5-large needs a 40GB GPU; mT5-xl needs roughly four times that memory. Code: github.com/ufal/crac2023-corpipe.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.crac-sharedtask.4 · arXiv 2311.14391

@inproceedings{straka2023ufal,
  title = {{ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution}},
  author = {Straka, Milan},
  booktitle = {Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution},
  year = {2023},
  doi = {10.18653/v1/2023.crac-sharedtask.4},
  eprint = {2311.14391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.crac-sharedtask.4}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations. find our card
  2. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti… find our card
  3. Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97–102, Dublin, Ireland. Association for Computational Linguistics. find
  4. Ondřej Pražák and Miloslav Konopik. 2022. End-to-end multilingual coreference resolution with mention head prediction. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 23–27, Gyeongju, Republic of Korea. Association for Computational Linguistics. find our card
  5. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  6. Ondřej Pražák, Miloslav Konopík, and Jakub Sido. 2021. Multilingual coreference resolution with harmonized annotations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1119–1123, Held Online. INCOMA Ltd. find our card
  7. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufma… find
  8. Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  9. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  10. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–… find our card
  11. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock-holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pag… find
  12. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models. arXiv e-prints, page arXiv:2301.10472. find
  13. Milan Straka and Jana Straková. 2022. ÚFAL CorPipe at CRAC 2022: Effectivity of multilingual models for coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 28–37, Gyeongju, Republic of Korea. Association for Computational Linguistics. find our card
  14. Michal Novák, Martin Popel, Zdeněk Žabokrtský, Daniel Zeman, Anna Nedoluzhko, Kutay Acar, Peter Bourgonje, Silvie Cinková, Gülşen Cebiroǧlu Eryiǧit, Jan Hajič, Christian Hardmeier, Dag Haug, Tollef Jørgensen, Andre Kåsen, Pauline Krielke, Frédéric Landragin, Ekaterina Lapshinova-Koltunski, Pett… find
  15. Sara Stymne, Miryam de Lhoneux, Aaron Smith, and Joakim Nivre. 2018. Parser training with heterogeneous treebanks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 619–625, Melbourne, Australia. Association for Computational L… find
  16. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826. find
  17. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations… find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  19. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. find
  20. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  21. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, and Daniel Zeman. 2023. Findings of the Second Shared Task on Multilingual Coreference Resolution. In Proceedings of the CRAC 2023 Shared Task on Multilingual Coreferenc… find our card
  22. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, Daniel Zeman, and Yilun Zhu. 2022. Findings of the shared task on multilingual coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Corefe… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/straka-2023-fal-corpipe-crac-2023-larger/