Elamur Research⌕
☰
home / papers / 2022 / NARC – Norwegian Anaphora Resolution Corpus

NARC – Norwegian Anaphora Resolution Corpus

Summary

Norwegian has had no openly available dataset for coreference resolution (CR). The earlier BREDT corpus exists but is not publicly released. The authors fill this gap with NARC. They add a document-level anaphora layer to the Norwegian Dependency Treebank (NDT), which already carries UD syntax and NorNE named entities. To speed up annotation, candidate markables were generated automatically from the treebank's part-of-speech and dependency information. Annotators treated these candidates as suggestions and followed guidelines adapted mainly from OntoNotes and BREDT. They linked mentions using three relation types and handled nested mentions, cataphora and compounds. This paper describes 326 Bokmål documents, while the Nynorsk portion is nearly finished. For a first benchmark, the authors trained the word-level wl-coref model of Dobrovolskii (2021) on top of Norwegian and multilingual transformers. The resulting scores are broadly in line with CR results for other languages, with MUC notably strong. Entity-level metrics were weaker, which the authors attribute to problems with assigning entities. Because NARC stays aligned with the treebank's other annotation layers, it is a rich, reusable resource for Norwegian NLP.

Key points

  • NARC builds on the NDT, which has about 300,000 tokens per written standard (around 600,000 in total); roughly 85% of the text is news, alongside government reports, parliamentary transcripts and blogs.
  • It uses three relations: COREFERENCE, BRIDGING and SPLIT-ANTECEDENT. Unlike OntoNotes, appositives get no separate relation and are instead included inside the markable span.
  • Four language models were compared: NorBERT2, NB-BERT, XLM-R base and mBERT. Their initial wl-coref F1 scores ranged from 51.3 to 56.1, and NorBERT2 and XLM-R were kept for fine-tuning on an 80% training split.
  • Evaluation used MUC, B³, CEAFe, CoNLL-F1 and LEA. MUC scores were high, but B³ and CEAFe were clearly lower.
  • The data and annotation guidelines are released at github.com/ltgoslo/NARC, and a Nynorsk portion following the same guidelines is planned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Treebanks

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mhlum2022narc,
  title = {{NARC – Norwegian Anaphora Resolution Corpus}},
  author = {Mæhlum, Petter and Haug, Dag and Jørgensen, Tollef and Kåsen, Andre and Nøklestad, Anders and Rønningstad, Egil and Solberg, Per Erik and Velldal, Erik and Øvrelid, Lilja},
  booktitle = {Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2022},
  doi = {10.48448/2k7a-ma68},
  url = {https://doi.org/10.48448/2k7a-ma68}
}

References

62 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oshin Agarwal, Sanjay Subramanian, Ani Nenkova, and Dan Roth. 2019. Evaluation of named entity coreference. In Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 1–7, Minneapolis, USA. Association for Computational Linguistics. find our card
  2. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405– 1415, Beijing, China. Association for Computational Linguistics. ↗ our card
  3. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the First International Conference on Language Resources and Evaluation, Workshop on Linguistics Coreference, pages 563–566, Granada, Spain. find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  5. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 44–54, Marseille, France. European Language Resources Association. ↗ our card
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised https://github.com/ltgoslo/NARC cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 229–238, Berlin, Germany. Association for Computational Linguistics. ↗
  7. Ralph Grishman and Beth M Sundheim. 1996. Message understanding conference-6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, pages 466– 471. find
  8. Aron Culotta, Michael Wick, and Andrew McCallum. 2007. First-order probabilistic models for coreference resolution. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 81–88, Rochester, New York. Association for Computational Linguistics. find our card
  9. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1108–1118, Denver, Colorado. Association for Computational Linguistics. find our card
  10. Pascal Denis and Jason Baldridge. 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 236–243, Rochester, New York. Association for Computational Linguistics. find our card
  11. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 conference on empirical methods in natural language processing, pages 1152–1161, Singapore. ACL and AFNLP. find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Iris Hendrickx, Gosse Bouma, Frederik Coppens, Walter Daelemans, Veronique Hoste, Geert Kloosterman, Anne-Marie Mineur, Joeri Van Der Vloet, and Jean-Luc Verschelde. 2008. A coreference corpus and resolution system for Dutch. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find our card
  14. Vladimir Dobrovolskii. 2021. Word-level coreference resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7670–7675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗ our card
  15. Gordana Ilić Holen. 2007. Automatic anaphora resolution for norwegian (ARN). In Anaphora: Analysis, Algorithms and Applications, pages 151–166, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  16. Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020. NorNE: Annotating named entities for Norwegian. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4547–4556, Marseille, France. European Language Resources Association. ↗
  17. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program – tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). ↗ our card
  18. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  19. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971–1982, Seattle, Washington, USA. Association for Computational Linguistics. ↗ our card
  20. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy. Association for Computational Linguistics. ↗ our card
  21. Jenny Rose Finkel and Christopher D Manning. 2008. Enforcing transitivity in coreference resolution. In Proceedings of ACL-08: HLT, Short Papers, pages 45–48, Columbus, Ohio, USA. Association for Computational Linguistics. find our card
  22. Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coreference resolution without span representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Abbas Ghaddar and Philippe Langlais. 2016. Coreference in Wikipedia: Main concept resolution. In Pro-Processing (Volume 2: Short Papers), pages 14–19, Online. Association for Computational Linguistics. find our card
  23. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 795–804, Jeju Island, Korea. Association for Computational Linguistics. find our card
  24. Mateusz Kopeć and Maciej Ogrodniczuk. 2014. Inter-annotator Agreement in Coreference Annotation of Polish, pages 149–158. Springer International Publishing, Cham. find
  25. Nafise Sadat Moosavi. 2020. Robustness in Coreference Resolution. Ph.D. thesis, Neuphilologische Fakultät > Institut für Computerlinguistik – Heidelberg University, Heidelberg. ↗
  26. Iorn Korzen and Matthias Buch-Kromann. 2011. Anaphoric relations in the copenhagen dependency treebanks. In Proceedings of DGfS Workshop, Göttingen, Germany, pages 83–98. find
  27. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  28. Per E Kummervold, Javier De la Rosa, Freddy Wetjen, and Svein Arne Brygfjeld. 2021. Operationalizing a national digital library: The case for a Norwegian transformer model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 20–29, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. ↗
  29. Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jiří Mírovský. 2016. Coreference in Prague Czech-English Dependency Treebank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 169–176, Portorož, Slovenia. European Language Resources Association (ELRA). find our card
  30. Andrey Kutuzov, Jeremy Barnes, Erik Velldal, Lilja Øvrelid, and Stephan Oepen. 2021. Large-scale contextualised language modelling for Norwegian. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 30–40, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. ↗
  31. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, Daniel Zeman, Peter Bourgonje, Silvie Cinková, Jan Hajič, Christian Hardmeier, Pauline Krielke, Frédéric Landragin, Ekaterina Lapshinova-Koltunski, M. Antònia Martí, Marie Mikulová, Maciej Ogrodniczuk, Marta Recasens, Manfred Stede, Milan Straka, Svetlana Toldova, Veronika Vincze, and Voldemaras Žitkus. 2022. Coreference in universal dependencies 1.0 (CorefUD 1.0). LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  32. Ekaterina Lapshinova-Koltunski, Christian Hardmeier, and Pauline Krielke. 2018. ParCorFull: a parallel corpus annotated with full coreference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  33. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. ↗ our card
  34. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 104– 111, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
  35. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  36. Anders Nøklestad. 2009. A machine learning approach to anaphora resolution including named entity recognition, PP attachment disambiguation, and animacy detection. Ph.D. thesis, University of Oslo. find
  37. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. ↗ our card
  38. Maciej Ogrodniczuk, Katarzyna Głowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2016. Polish coreference corpus. In Human Language Technology. Challenges for Computer Science and Linguistics, pages 215–226, Cham. Springer International Publishing. find
  39. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 135–142, Barcelona, Spain. find our card
  40. Lilja Øvrelid and Petter Hohle. 2016. Universal Dependencies for Norwegian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1579–1585, Portorož, Slovenia. European Language Resources Association (ELRA). Norwegian dependency treebank. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 789–795, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  41. Hoifung Poon and Pedro Domingos. 2008. Joint unsupervised coreference resolution with Markov Logic. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 650–659, Honolulu, Hawaii. Association for Computational Linguistics. find our card
  42. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102–107, Avignon, France. Association for Computational Linguistics. find
  43. Martin Popel, Zdeněk Žabokrtskỳ, Anna Nedoluzhko, Michal Novák, and Daniel Zeman. 2021. Do UD trees match mention spans in coreference annotations? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3570–3576. Association for Computational Linguistics. find our card
  44. Rhea Sukthanker, Soujanya Poria, Erik Cambria, and Ramkumar Thirunavukarasu. 2020. Anaphora and coreference resolution: A review. Information Fusion, 59:139–162. ↗
  45. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
  46. Svetlana Toldova and Max Ionov. 2017. Coreference resolution for Russian: the impact of semantic features. Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2017”, pages 339–348. find
  47. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  48. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Natural Language Engineering, 26(1):95–128. ↗
  49. Erik Velldal, Lilja Øvrelid, and Petter Hohle. 2017. Joint UD parsing of Norwegian Bokmål and Nynorsk. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 1–10, Gothenburg, Sweden. Association for Computational Linguistics. find
  50. Marta Recasens, Lluís Màrquez, Emili Sapena, M. Antònia Martí, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1–8, Uppsala, Sweden. Association for Computational Linguistics. find our card
  51. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference (MUC-6), Columbia, Maryland. find our card
  52. Marta Recasens and M Antònia Martí. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language resources and evaluation, 44(4):315–345. ↗
  53. Veronika Vincze, Klára Hegedűs, Alex Sliz-Nagy, and Richárd Farkas. 2018. SzegedKoref: A Hungarian coreference corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  54. Ina Roesiger, Arndt Riester, and Jonas Kuhn. 2018. Bridging resolution: Task definition, corpus resources and rule-based experiments. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3516–3528, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  55. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗ our card
  56. Ina Rösiger. 2018. BASHI: A corpus of Wall Street Journal articles annotated with bridging links. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗ our card
  57. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer, 3(3):3–4. find
  58. Per Erik Solberg, Arne Skjærholt, Lilja Øvrelid, Kristin Hagen, and Janne Bondi Johannessen. 2014. find
  59. The Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics. find our card
  60. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533, Online. Association for Computational Linguistics. ↗ our card
  61. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  62. Voldemaras Žitkus and Rita Butkiene. 2018. Coreference annotation scheme and corpus for Lithuanian language. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 243–250. IEEE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hlum-2022-narc-norwegian-anaphora-resolution-corpus/