Elamur Research⌕
☰
home / papers / 2023 / A Sequence-to-Sequence Approach for Arabic Pronoun Resolution

A Sequence-to-Sequence Approach for Arabic Pronoun Resolution

Summary

The paper addresses pronoun resolution in Arabic: deciding which earlier noun or noun phrase a pronoun points to. The authors note that prior Arabic work relied mostly on handcrafted linguistic features and classic classifiers. They recast the task as sequence-to-sequence labelling. A pre-trained AraBERT encoder reads the text, and a Bi-LSTM decoder assigns every input token a probability of belonging to the true antecedent. Training uses a binary target sequence that marks the antecedent tokens. Experiments run on the AnATAr corpus, and the model is compared against KNN, logistic regression, SVM and feature-engineered systems. Three refinements are tested:

  • appending the anaphor's text to the paragraph,
  • masking the output so only candidate positions are scored,
  • discarding candidates that disagree with the pronoun in gender or number.

Together these lift results to roughly 81% MRR and 71% F1, and precision, recall and accuracy also improve. The work shows that pretrained neural encoders can replace manual feature design for a morphologically rich, under-resourced language.

Key points

  • Architecture: AraBERT encoder plus Bi-LSTM decoder, producing a per-token antecedent score against a binary gold sequence.
  • Data: the AnATAr corpus (Hammami et al., 2009), described as the first annotated Arabic anaphora corpus. Other resources surveyed include QurAna (2012, built on the Quran) and A3C (2021, covering pronominal and verbal anaphora with gender, number and frequency information).
  • Baselines: KNN, logistic regression, SVM and handcrafted-feature models. The neural model beats all of them on every metric reported.
  • Best configuration reaches about 81% MRR and 71% F1, using anaphor concatenation, a candidate mask and gender/number agreement filtering.
  • Historical context: an early multilingual approach adapted from English reported about 95% success on 190 Arabic anaphors. Elghamry et al. (2007) reported 87.6% on a 5000-pronoun gold set using gender, number and rationality cues.

Interesting ideas

  • I1. Rather than classifying each pronoun–candidate pair separately, the model tags every token of the passage as inside or outside the antecedent, so one pass scores all options and span boundaries come out naturally. — For Mova (frames): coreference in prag could score antecedent spans over the UD-tokenized passage in a single sweep, and the per-token scores remain inspectable as an explanation.
  • I2. The pronoun being resolved is attached to the passage input, telling the encoder which reference is in question without changing the architecture. — For Mova (frames): when several pronouns occur in a sentence, a query-marked input lets one resolver handle each pronoun separately while sharing the same passage encoding.
  • I3. A mask limits output scoring to legitimate candidate positions, so probability mass is not spent on tokens that can never be antecedents. — For Mova (frames): candidate NPs taken from the UD tree (nominal heads and their subtrees) could define the mask, giving the resolver a structural prior instead of letting it guess over raw tokens.
  • I4. A hard morphological agreement filter (gender, number) prunes candidates before neural scoring, combining symbolic rules with learned ranking. — For Mova (tokens): morphology features from crate en (number, plus gender for English pronouns) can drive an explicit, logged pruning step whose rejections are explainable.
  • I5. The authors report MRR next to F1, which treats resolution as ranking and gives partial credit when the right antecedent is near the top. — For Mova (learn): adding MRR to coreference evaluation would show whether silver-data or teacher-tier training improves candidate ordering even when the top-1 choice is still wrong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.11529 · arXiv 2305.11529

@misc{murayshid2023sequence,
  title = {{A Sequence-to-Sequence Approach for Arabic Pronoun Resolution}},
  author = {Murayshid, Hanan S. and Benhidour, Hafida and Kerrache, Said},
  year = {2023},
  doi = {10.48550/arxiv.2305.11529},
  eprint = {2305.11529},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.11529}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. W. Antoun, F. Baly, and H. Hajj, “Arabert: Transformer-based model for arabic language understanding,” in LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, 2020, p. 9. find our card
  2. A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural Networks, vol. 18, no. 5, pp. 602–610, 2005, iJCNN 2005. find
  3. S. Hammami, L. Belguith, and A. Ben Hamadou, “Arabic Anaphora Resolution: Corpora Annotation with Coreferential Links,” Int. Arab J. Inf. Technol., vol. 6, pp. 480–488, Nov. 2009. find
  4. R. Sukthanker, S. Poria, E. Cambria, and R. Thirunavukarasu, “Anaphora and coreference resolution: A review,” Information Fusion, 5 2018. [Online]. Available: http://arxiv.org/abs/1805.11824 ↗
  5. V. Nemcik, “Anaphora resolution,” Master’s thesis, Masaryk University, 2006. find
  6. M. A. K. Halliday and R. Hasan, “Review of COHESION IN ENGLISH,” JSTOR, vol. 14, no. 1, pp. 47–50, 1976, publisher: Penn State University Press. [Online]. Available: https://www.jstor.org/stable/42945277 ↗
  7. S. Bouzid and F. Trabelsi, “How to combine salience factors for arabic pronoun anaphora resolution,” in ACS/IEEE International Conference on Computer Systems and Applications, 2017, pp. 929– 936. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/8308389/ ↗
  8. S. Mathlouthi Bouzid and C. Ben Othmane Zribi, “Aggregation of Word Embedding and Q-learning for Arabic Anaphora Resolution,” in Communications in Computer and Information Science, K. Smaïli, Ed., vol. 1108. Cham: Springer International Publishing, 2019, pp. 93–107, book Title: Arabic Language Processing: From Theory to Practice Series Title: Communications in Computer and Information Science. find
  9. A. Abolohom and N. Omar, “A Hybrid Approach to Pronominal Anaphora Resolution in Arabic,” Journal of Computer Science, vol. 11, no. 5, pp. 764–771, May 2015. [Online]. Available: http://thescipub.com/abstract/10.3844/jcssp.2015.764.771 ↗
  10. M. Schiehlen, “Optimizing algorithms for pronoun resolution,” in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. Geneva, Switzerland: COLING, aug 23–aug 27 2004, pp. 515–521. [Online]. Available: https://aclanthology.org/C04-1074 ↗
  11. R. Mitkov, L. Belguith, and M. Stys, “Multilingual robust anaphora resolution,” in Proceedings of the Third Conference on Empirical Methods for Natural Language Processing. Palacio de Exposiciones y Congresos, Granada, Spain: Association for Computational Linguistics, Jun. 1998, pp. 7–16. [Online]. Available: https://aclanthology.org/W98-1502 ↗ our card
  12. A.-B. Sharaf and E. Atwell, “QurAna: Corpus of the Quran annotated with pronominal anaphora,” in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12). Istanbul, Turkey: European Language Resources Association (ELRA), May 2012, pp. 130–137. find
  13. K. Seddik, A. Farghaly, and A. Fahmy, “Arabic Anaphora Resolution: Corpus of the Holy Qur’an Annotated with Anaphoric Information,” International Journal of Computer Applications, vol. 124, no. 15, pp. 35–43, Aug. 2015. [Online]. Available: http://www.ijcaonline.org/research/volume124/number15/seddik-2015-ijca-905709.pdf ↗
  14. K. Elghamry, R. Sabbagh, and N. El-Zeiny, “Arabic Anaphora Resolution Using the Web as Corpus,” Proceedings of The 7th . . . , Jan. 2007. find
  15. A. Abolohom and N. Omar, “A Machine Learning Approach to Anaphora Resolution in Arabic,” International Review on Computers and Software (IRECOS), vol. 9, no. 12, p. 1956, Dec. 2014. find
  16. S. Mathlouthi Bouzid, f. ben fraj trabelsi, and C. Zribi, “A novel approach based on reinforcement learning for anaphora resolution,” in International Business Information MAnagement conference, Oct. 2016. find
  17. S. Bouzid and C. Zribi, “A generic approach for pronominal anaphora and zero anaphora resolution in arabic language,” Procedia Computer Science, vol. 176, pp. 642–652, 2020. find
  18. F. B. F. Trabelsi, C. Ben Othmane Zribi, and S. Mathlouthi, “Arabic Anaphora Resolution Using Markov Decision Process,” in CI-CLing 2016: Computational Linguistics and Intelligent Text Processing, A. Gelbukh, Ed., vol. 9623. Cham: Springer International Publishing, 2018, pp. 520–532, book Title: Computational Linguistics and Intelligent Text Processing Series Title: Lecture Notes in Computer Science. find
  19. A. Abolohom and N. Omar, “A Computational Model for Resolving Arabic Anaphora using Linguistic Criteria,” Indian Journal of Science and Technology, vol. 10, no. 3, Jan. 2017. [Online]. Available: https://indjst.org/articles/a-computational-model-for-resolving-arabic-anaphora-using-linguistic-criteria ↗
  20. A. Abolohom, N. Omar, S. Pais, and J. Cordeiro, “A Comparative Study of Linguistic and Computational Features Based on a Machine Learning for Arabic Anaphora Resolution,” in Procedia Computer Science, vol. 189, 2021, pp. 37–47, iSSN: 18770509 Journal Abbreviation: Procedia Computer Science. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1877050921011613 ↗
  21. M. A. Chéragui, A. H. Dahou, and M. Abdelmoazz, “A3C: Arabic Anaphora Annotated Corpus,” in Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021), 2021. find
  22. C. Chen and V. Ng, “Chinese zero pronoun resolution with deep neural networks,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 778–788. [Online]. Available: https://aclanthology.org/P16-1074 ↗ our card
  23. R. Iida, K. Torisawa, J.-H. Oh, C. Kruengkrai, and J. Kloetzer, “Intra-sentential subject zero anaphora resolution using multi-column convolutional neural network,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1244–1254. [Online]. Available: https://aclanthology.org/D16-1132 ↗ our card
  24. Q. Yin, Y. Zhang, W. Zhang, T. Liu, and W. Y. Wang, “Zero pronoun resolution with attention-based neural network,” in Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA: Association for Computational Linguistics, Aug. 2018, pp. 13–23. [Online]. Available: https://aclanthology.org/C18-1002 ↗ our card
  25. V. Annam, N. Koditala, and R. Mamidi, “Anaphora resolution in dialogue systems for south asian languages,” CoRR, vol. abs/1911.09994, 2019. [Online]. Available: http://arxiv.org/abs/1911.09994 ↗
  26. Q. Yang, L. Yu, S. Tian, and J. Song, “Multi-attention-based capsule network for uyghur personal pronouns resolution,” IEEE Access, vol. 8, pp. 76 832–76 840, 2020. find
  27. S. Li, W. Li, C. Cook, C. Zhu, and Y. Gao, “Independently recurrent neural network (indrnn): Building A longer and deeper RNN,” CoRR, vol. abs/1803.04831, 2018. [Online]. Available: http://arxiv.org/abs/1803.04831 ↗
  28. F. Liu, L. Zettlemoyer, and J. Eisenstein, “The referential reader: A recurrent entity network for anaphora resolution,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5918–5925. [Online]. Available: https://aclanthology.org/P19-1593 ↗ our card
  29. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1724–1734. [Online]. Available: https://aclanthology.org/D14-1179 ↗
  30. K. Webster, M. Recasens, V. Axelrod, and J. Baldridge, “Mind the gap: A balanced corpus of gendered ambiguou,” in Transactions of the ACL, 2018, p. to appear. find
  31. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423 ↗
  32. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 3104–3112. find
  33. K. P. Murphy, Machine learning: a probabilistic perspective, ser. Adaptive computation and machine learning series. Cambridge, MA: MIT, 2012. find
  34. M. Beseiso and A. Al-Alwani, “A Coreference Resolution Approach using Morphological Features in Arabic,” International Journal of Advanced Computer Science and Applications, vol. 7, no. 10, 2016. find
  35. O. Obeid, N. Zalmout, S. Khalifa, D. Taji, M. Oudah, B. Alhafni, G. Inoue, F. Eryani, A. Erdmann, and N. Habash, “CAMeL tools: An open source python toolkit for Arabic natural language processing,” in Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 7022–7032. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.868 ↗ our card
  36. K. Toutanova, D. Klein, C. D. Manning, and Y. Singer, “Feature-rich part-of-speech tagging with a cyclic dependency network,” in Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, ser. NAACL ’03. USA: Association for Computational Linguistics, 2003, p. 173–180. [Online]. Available: https://doi.org/10.3115/1073445.1073478 ↗ our card
  37. T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21–27, 1967. find
  38. C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, 1995. [Online]. Available: https://doi.org/10.1023/A:1022627411411 ↗
  39. D. R. Cox, “The regression analysis of binary sequences,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 20, no. 2, pp. 215–232, 1958. find
  40. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. find
  41. S. van den Berg, Analysing Data using Linear Models, 5th ed. Netherlands: University of Twente, Jan. 2021. find
  42. J. Daniel and J. Martin, “Question answering and information retrieval,” in Speech and Language Processing, 3rd ed., 2023, ch. 14, pp. 269–295. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/murayshid-2023-sequence-sequence-approach-arabic-pronoun/