Elamur Research⌕
☰
home / papers / 2025 / Enhanced Arabic Text Retrieval with Attentive Relevance Scoring

Enhanced Arabic Text Retrieval with Attentive Relevance Scoring

Summary

Arabic is hard for retrieval systems. Its morphology is rich, diacritics are optional, and Modern Standard Arabic sits alongside many dialects. Dedicated dense retrievers for the language are also scarce. Prior work such as AraDPR showed that contrastively trained Arabic retrievers outperform both sparse methods and multilingual models. However, AraDPR and similar systems still rank passages by a plain dot product or cosine between two vectors. The authors argue that this final step is too crude for Arabic. They propose Adaptive Passage Retrieval (APR), a dual-encoder system initialised from the compact Arabic MiniBERT. APR replaces the fixed similarity with a small trainable module called Attentive Relevance Scoring (ARS). ARS projects the question and passage embeddings into a shared space, multiplies them element by element, applies tanh, and turns the result into a single sigmoid score through a learned attention vector. Passage vectors can still be computed ahead of time. The authors report clearly better ranking accuracy on Arabic question answering, although the result tables were not in the excerpt we received. They release their code publicly.

Key points

  • APR is a DPR-style dual encoder. The separate question encoder and passage encoder are both initialised from MiniBERT, an Arabic-pretrained model from Safaya et al.
  • The global representation of each text is its final-layer [CLS] vector, ℓ2-normalised onto the unit hypersphere to stabilise contrastive similarity.
  • ARS computes a score in four steps:
    • It projects the query and passage with matrices W_q and W_p (h×d).
    • It takes their element-wise product.
    • It applies tanh.
    • It combines the result with a learned vector w_a and passes it through a sigmoid, giving a score between 0 and 1.
  • At inference time, embeddings for all passages are precomputed. Only the query is encoded live, and ARS then scores the query against each stored passage vector.
  • The main reference point is AraDPR, which uses an AraBERT backbone and was trained on translated plus native Arabic QA data such as ARCD. Exact result numbers were not available in the provided text.

Interesting ideas

  • I1. Keep both sides encoded independently, and swap the fixed dot product for a tiny learned interaction head. Retrieval stays cacheable, but relevance is no longer limited to raw geometric closeness. — For Mova (answer): A QA reader could rank candidate story sentences for a question with such a head over frame-feature vectors. Because the features are named slots rather than opaque dimensions, each component of the element-wise product can be shown as an explanation.
  • I2. Bound the relevance score to (0,1) with a sigmoid instead of leaving it as an unbounded similarity. This gives an absolute value that can be thresholded, not just a ranking. — For Mova (answer): The fable and school-test readers could abstain or say "not stated in the text" when the best supporting fact falls below a tuned threshold, instead of always returning the top candidate.
  • I3. Put the modelling capacity into a cheap scoring layer on top of a lightweight encoder, rather than scaling up the backbone. This suits low-resource or CPU settings. — For Mova (frames): Coreference could store one vector per entity and score each new mention with a small projected-product-sigmoid head. Antecedent search would stay fast, and the per-feature contributions could be logged.
  • I4. Normalise vectors to the unit sphere before the learned interaction. This keeps contrastive training stable, because magnitude cannot dominate the score. — For Mova (learn): When training any pairwise matcher, such as aligning silver frames between witness parsers, normalising inputs first could make agreement scores comparable across sentences and parsers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages · Information retrieval & search

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.23404 · arXiv 2507.23404

@misc{bekhouche2025enhanced,
  title = {{Enhanced Arabic Text Retrieval with Attentive Relevance Scoring}},
  author = {Bekhouche, Salah Eddine and Benlamoudi, Azeddine and Bounab, Yazid and Dornaika, Fadi and Hadid, Abdenour},
  year = {2025},
  doi = {10.48550/arxiv.2507.23404},
  eprint = {2507.23404},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.23404}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih, “Dense passage retrieval for open-domain question answering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, pp. 6769–6781, Association for Computational Linguistics. find our card
  2. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt, “Arabicaqa: A comprehensive dataset for arabic question answering,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 2049–2059. find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 4171–4186, Association for Computational Linguistics. find
  4. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. find
  5. Jeff Johnson, Matthijs Douze, and Hervé Jégou, “Billion-scale similarity search with gpus,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019. find our card
  6. Stephen Robertson, Hugo Zaragoza, et al., “The probabilistic relevance framework: Bm25 and beyond,” Foundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. find
  7. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Lau, Jimmy Lin, and Paul Bennett, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” 2020. find our card
  8. Yingqi Qu, Yujing Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang, “RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, pp. 5835–5847, Association for Computational Linguistics. find our card
  9. Omar Khattab and Matei Zaharia, “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, 2020, pp. 39–48. find our card
  10. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia, “Colbertv2: Effective and efficient retrieval via lightweight late interaction,” arXiv preprint arXiv:2112.01488, 2021. ↗ our card
  11. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant, “Splade: Sparse lexical and expansion model for first stage ranking,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. find our card
  12. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. ↗
  13. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu, “Tinybert: Distilling bert for natural language understanding,” arXiv preprint arXiv:1909.10351, 2019. ↗
  14. Ahmed Magdy Ezzeldin and Mohamed Shaheen, “A survey of arabic question answering: challenges, tasks, approaches, tools, and future trends,” in Proceedings of The 13th international Arab conference on information technology (ACIT 2012), 2012, pp. 1–8. find
  15. Wissam Antoun, Fady Baly, and Hazem Hajj, “Arabert: Transformer-based model for arabic language understanding,” arXiv preprint arXiv:2003.00104, 2020. ↗ our card
  16. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi, “Arbert & marbert: Deep bidirectional transformers for arabic,” arXiv preprint arXiv:2101.01785, 2020. ↗ our card
  17. Wissam Antoun, Fady Baly, and Hazem Hajj, “Araelectra: Pre-training text discriminators for arabic language understanding,” arXiv preprint arXiv:2012.15516, 2020. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash, “The interplay of variant, size, and task type in arabic pre-trained language models,” arXiv preprint arXiv:2103.06678, 2021. ↗ our card
  19. Kareem Darwish, Walid Magdy, et al., “Arabic information retrieval,” Foundations and Trends® in Information Retrieval, vol. 7, no. 4, pp. 239–342, 2014. find
  20. Rasha Obeidat, Marwa Al-Harbi, Mahmoud Al-Ayyoub, and Luay Alawneh, “Arquad: An expert-annotated arabic machine reading comprehension dataset,” Cognitive Computation, vol. 16, no. 3, pp. 984–1003, 2024. find
  21. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, “Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 454– 470, 2020. find
  22. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck, “Learning deep structured semantic models for web search using clickthrough data,” in Proceedings of the 22nd ACM international conference on Information & Knowledge Management, 2013, pp. 2333–2338. find
  23. Sparsh Gupta and Vitor R Carvalho, “Faq retrieval using attentive matching,” in Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 929–932. find
  24. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih, “Dense passage retrieval for open-domain question answering.,” in EMNLP (1), 2020, pp. 6769–6781. find our card
  25. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret, “Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media,” arXiv preprint arXiv:2007.13184, 2020. ↗ our card
  26. Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR), 2019. find
  27. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj, “Neural arabic question answering,” arXiv preprint arXiv:1906.05394, 2019. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bekhouche-2025-enhanced-arabic-text-retrieval-attentive/