Elamur Research⌕
☰
home / papers / 2020 / Neural Coreference Resolution for Arabic

Neural Coreference Resolution for Arabic

Summary

Arabic coreference resolution has barely advanced since the CoNLL 2012 shared task. The strongest system was still the feature-based resolver of Björkelund and Kuhn (2014), and no neural model had been tried. The authors fill this gap by adapting a leading English end-to-end architecture to Arabic. Their baseline is the Lee et al. (2018) model with BERT embeddings instead of ELMo, trained and tested on the Arabic part of OntoNotes 5.0 (CoNLL 2012 data). They add three improvements on top of it:

  • rule-based normalization of letter variants and removal of diacritics;
  • the monolingual AraBERT in place of multilingual BERT;
  • a separately trained mention detector, combined with an "end-to-end annealing" schedule that mixes joint and pipeline training.

The complete system reaches 63.9% CoNLL F1, which is 15.2 points above the previous best. This is the first neural resolver built specifically for Arabic, and its code is public. It shows that modern architectures work even with a small training corpus. The remaining obstacles are scarce annotated data and the lack of an Arabic BERT-large.

Key points

  • Code is released at github.com/juntaoy/aracoref. The model builds on Lee et al. (2018) with BERT embeddings in the style of Kantor and Globerson (2019a), plus fastText and character CNN features.
  • Normalization and diacritic stripping alone add about 7 CoNLL F1 points, mainly by raising recall through reduced sparsity.
  • Replacing multilingual BERT with AraBERT adds roughly 4.7–4.8 points for coreference and 2.4 points for mention detection.
  • The external mention detector with end-to-end annealing adds 0.8 points and raises mention recall by 2.3 points. The final score is 63.9% CoNLL F1, which is +15.2 over Björkelund and Kuhn (2014).
  • The Arabic portion of OntoNotes is small: about 300,000 tokens in 447 documents. ACE has about 500,000 Arabic tokens but covers only seven entity types.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2011.00286

@inproceedings{aloraini2020neural,
  title = {{Neural Coreference Resolution for Arabic}},
  author = {Aloraini, Abdulrahman and Yu, Juntao and Poesio, Massimo},
  booktitle = {Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2020},
  eprint = {2011.00286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.00286}
}

References

51 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullatif Abolohom and Nazlia Omar. 2015. A hybrid approach to pronominal anaphora resolution in arabic. Journal of Computer Science, 11(5):764. ↗
  2. Abdulrahman Aloraini and Massimo Poesio. 2020. Cross-lingual zero pronoun resolution. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 90–98. find our card
  3. Maha Althobaiti, Udo Kruschwitz, and Massimo Poesio. 2014. Aranlp: A java-based library for the processing of arabic text. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗ our card
  5. Majdi Beseiso and Abdulkareem Al-Alwani. 2016. A coreference resolution approach using morphological features in arabic. International Journal of Advanced Computer Science and Applications, 7(10):107–113. ↗
  6. Anders Björkelund and Pierre Nugues. 2011. Exploring lexicalized features for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 45–50. find
  7. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47–57. find our card
  8. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
  9. Saoussen Mathlouthi Bouzid, Fériel Ben Fraj Trabelsi, and Chiraz Ben Othmane Zribi. 2017. How to combine salience factors for arabic pronoun anaphora resolution. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pages 929–936. IEEE. find
  10. Chen Chen and Vincent Ng. 2012. Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 56–63. find our card
  11. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Association for Computational Linguistics (ACL). find our card
  12. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Empirical Methods on Natural Language Processing (EMNLP). find our card
  13. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Association for Computational Linguistics (ACL). find our card
  14. Walter Daelemans, Jakub Zavrel, Kurt Van Der Sloot, and Antal Van den Bosch. 2004. Timbl: Tilburg memory-based learner. Tilburg University. find
  15. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074. find
  16. H. Daume and D. Marcu. 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In Proc. HLT/EMNLP, Vancouver. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  18. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec, volume 2, pages 837–840. Lisbon. find our card
  19. Timothy Dozat and Christopher Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of 5th International Conference on Learning Representations (ICLR). find our card
  20. Eraldo Fernandes, Cícero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 41–48, Jeju Island, Korea, July. Association for Computational Linguistics. find our card
  21. Eraldo Rezende Fernandes, Cícero Nogueira dos Santos, and Ruy Milidiú. 2014. Latent trees for coreference resolution. In Computational Linguistics, 40(4), pages 801–835. ↗ our card
  22. Nizar Habash and Fatiha Sadat. 2006. Arabic preprocessing schemes for statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 49–52. find
  23. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy. find our card
  24. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019a. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. ↗ our card
  25. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019b. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China, November. Association for Computational Linguistics. find our card
  26. Ben Kantor and Amir Globerson. 2019a. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy, July. Association for Computational Linguistics. find our card
  27. Ben Kantor and Amir Globerson. 2019b. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677. find our card
  28. Yuri Kuratov and Mikhail Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint arXiv:1905.07213. ↗
  29. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In CONLL Shared Task ’11 Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 28–34. find our card
  30. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  31. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. Proceedings of the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  32. Sangah Lee, Hansol Jang, Yunmee Baik, Suzi Park, and Hyopil Shin. 2020. Kr-bert: A small-scale korean-specific language model. arXiv preprint arXiv:2008.03979. ↗
  33. Baoli li. 2012. Learning to model multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL2012-Shared Task. find
  34. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. ↗ our card
  35. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  36. M. Poesio, R. Stuckardt, and Y. Versley. 2016. Anaphora Resolution: Algorithms, Resources and Applications. Springer, Berlin. find
  37. Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets. In CLiC-it. find
  38. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task. Association for Computational Linguistics, Association for Computational Linguistics., pages 1–40. find our card
  39. Wee M. Soon, Daniel C. Y. Lim, and Hwee T. Ng. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4), December. ↗ our card
  40. Fabio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2019. Portuguese named entity recognition using bert-crf. arXiv preprint arXiv:1909.10649. ↗
  41. Marcus Stamborg, Dennis Medved, Peter Exner, and Pierre Nugues. 2012. Using syntactic dependencies to solve coreferences. In Joint Conference on EMNLP and CoNLL2012-Shared Task. find our card
  42. Fériel Ben Fraj Trabelsi, Chiraz Ben Othmane Zribi, and Saoussen Mathlouthi. 2016. Arabic anaphora resolution using markov decision process. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 520–532. Springer. find
  43. Olga Uryupina, Alessandro Moschitti, and Massimo Poesio. 2012. Bart goes multilingual: the unitn/essex submission to the conll-2012 shared task. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 122–128. find
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  45. Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. Bart: A modular toolkit for coreference resolution. In Proceedings of the ACL-08: HLT Demo Session, pages 9–12. find our card
  46. Sam Wiseman, Alexander M Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1416–1426. find our card
  47. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004. find our card
  48. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online, July. Association for Computational Linguistics. find our card
  49. Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020a. Neural mention detection. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1–10, Marseille, France, May. European Language Resources Association. ↗
  50. Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020b. A cluster ranking model for full anaphora resolution. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 11–20, Marseille, France, May. European Language Resources Association. ↗ our card
  51. Desislava Zhekova and Sandra Kübler. 2010. Ubiu: A language-independent system for coreference resolution. In Proceedings of the 5th International Workshop on Semantic Evaluation, page 96–99. find our card

↑ back to top

This page in Markdown →