Elamur Research⌕
☰
home / papers / 2025 / Evaluating RAG Pipelines for Arabic Lexical Information Retrieval: A Comparative Study of Embedding and Generation Models

Evaluating RAG Pipelines for Arabic Lexical Information Retrieval: A Comparative Study of Embedding and Generation Models

Paper · 2025 · Raghad Al-Rasheed, Abdullah Al Muaddi, Hawra Aljasim et al.

Summary

The paper asks how well a retrieval-augmented generation (RAG) pipeline can answer questions about Arabic words, a setting where rich morphology and diacritics make retrieval and generation difficult. The authors index entries from the Riyadh dictionary, which covers more than 88,000 words. Entries are split at the paragraph level and stored in FAISS. They then run two comparisons. First, they test retrievers built two ways: mean-pooled token embeddings from AraBERT v2, CAMeLBERT and AraELECTRA, and dedicated sentence encoders (E5-large, BGE, Arabic-NLI-Matryoshka). Second, they pass the retrieved context to six LLMs and score their answers. Sentence-level encoders retrieve far better than pooled word embeddings, and E5 performs best. Among generators, the GPT-4 family (reported as GPT-4o in the body) leads. A breakdown by lexical task shows that POS tagging and writing usage examples remain weak for every model. The study gives Arabic-script NLP a practical reference for choosing pipeline components when building dictionary-backed question answering.

Key points

  • Dictionary text is chunked by paragraph with a cap of 512 tokens. Longer paragraphs are cut into windows that overlap by 50 tokens. Vectors go into a FAISS IndexFlatIP index, scored by cosine similarity.
  • E5 reached a Top-5 Recall of 0.88 but an MRR of only 0.48. This means the correct entry is usually retrieved but often not ranked first. Mean-pooled AraBERT v2, CAMeLBERT and AraELECTRA trailed the sentence encoders.
  • The generators tested were GPT-4/GPT-4o, GPT-3.5, Gemini-1.5 Flash, SILMA-9B-Instruct, Aya 8B and AceGPT-13B, all run at temperature 0. GPT-4 scored F1 0.90, accuracy 0.82 and cosine similarity 0.87.
  • Results were broken down across eight lexical task types:
    • Morphological pattern identification was easy, with GPT-4o at F1 0.98 and accuracy 0.96.
    • Diacritization separated GPT-4o from the other models.
    • Example generation was the hardest task, where the best model reached only F1 0.50 and accuracy 0.33.
  • The authors note that the evaluation set is unbalanced across task types. They also sketch how the pipeline could carry over to Pashto, Sindhi and Uyghur.

Interesting ideas

  • I1. Reporting Recall@1/3/5 together with MRR exposes the difference between "the answer is somewhere in the candidates" and "the answer is ranked first," which a single number would hide.
  • I2. Splitting a lexical QA benchmark into question types (diacritics, morphological pattern, POS, semantic relation, translation, example generation) reveals that some types are easy and others nearly fail, even with the same model.
  • I3. The finding that LLMs handle pattern and relation questions well but are unreliable on POS and example writing tells us which jobs an LLM is fit to do.
  • I4. Chunking a dictionary along entry/paragraph boundaries, and overlapping windows only when an entry is too long, keeps each retrieved unit self-contained in meaning.
  • I5. Running every generator at temperature 0 makes the model comparison reproducible and removes sampling noise from the scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): When Mova looks up candidate senses or ontology entries in WordNet/FrameNet/Wikidata, it could log recall-at-k and reciprocal rank separately. This would show whether errors come from missing candidates or from poor ranking.
  • I2 (learn): Evaluation reports could be split by question type, so that a strong average cannot mask a stage that consistently breaks on one kind of question.
  • I3 (learn): LLM teacher tiers could be trusted for relation or sense judgements but not as a source of silver POS labels. Those labels would still come from agreement among witness parsers.
  • I4 (world): Compiled lexical knowledge (verb classes, WordNet glosses) could be stored and retrieved one entry at a time, so every fact pulled into the world of a text has a clear, explainable source.
  • I5 (learn): Calls to the LLM teacher could use deterministic decoding, so silver data and negative-control runs can be regenerated exactly and differences traced to a specific change.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Raghad Al-Rasheed, Abdullah Al Muaddi, Hawra Aljasim, Rawan Al-Matham, Muneera Alhoshan, Asma Al Wazrah and Abdulrahman AlOsaimy. 2025. Evaluating RAG Pipelines for Arabic Lexical Information Retrieval: A Comparative Study of Embedding and Generation Models. Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alrasheed2025evaluating,
  title = {{Evaluating RAG Pipelines for Arabic Lexical Information Retrieval: A Comparative Study of Embedding and Generation Models}},
  author = {Raghad Al-Rasheed, Abdullah {Al Muaddi}, Hawra Aljasim, Rawan Al-Matham, Muneera Alhoshan, Asma {Al Wazrah} and Abdulrahman AlOsaimy},
  booktitle = {Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script},
  year = {2025},
  url = {https://aclanthology.org/2025.abjadnlp-1.16/}
}

References

20 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, and Ahmed Ali. 2024. Larabench: Benchmarking arabic ai with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. ↗
  2. SILMA AI. 2023. Silma-9b-instruct-v1.0. https://huggingface.co/silma-ai/ SILMA-9B-Instruct-v1.0. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020a. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020b. Araelectra: Pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516. ↗
  5. Vashista Aryabumi, James Dang, Dhanusha Talupuru, Sarvesh Dash, Daniel Cairuz, Harrison Lin, et al. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032. ↗
  6. Laura Caspari, Kanishka Ghosh Dastidar, Saber Zerhoudi, Jelena Mitrovic, and Michael Granitzer. 2024. Beyond benchmarks: Evaluating embedding model similarity for retrieval augmented generation systems. arXiv preprint arXiv:2407.08275. ↗
  7. Nadezhda Chirkova, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. 2024. Retrieval-augmented generation in multilingual settings. arXiv preprint arXiv:2407.01463. ↗
  8. Samhaa R. El-Beltagy and Mohamed A. Abdallah. 2024. Exploring retrieval augmented generation in arabic. In Procedia Computer Science.
  9. Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, et al. 2024. Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. arXiv preprint arXiv:2408.07983. llama-3 on arabic language generation tasks. In Proceedings of The Second Arabic Natural Language Processing Conference. ↗
  10. Ali Mahboub, Muhy Eddin Za’ter, et al. 2024. Evaluation of semantic search and its role in retrieved-augmented-generation (rag) for arabic language. arXiv preprint arXiv:2403.18350. ↗
  11. Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019. Jointly measuring diversity and quality in text generation models. arXiv preprint arXiv:1904.03971. ↗
  12. OpenAI. 2023. GPT-4 Technical Report. Technical report. ↗
  13. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR.
  14. Machel Reid, Nikolay Savinov, Dmitry Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-Baptiste Alayrac, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. ↗
  15. Spurthi Setty, Harsh Thakkar, Alyssa Lee, Eden Chung, and Natan Vidra. 2024. Improving retrieval for rag based question answering models on financial documents. arXiv preprint arXiv:2404.07221. ↗
  16. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Ranjan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. ↗
  17. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. A Model Responses for different Task Haoyang Huang, Feng Yu, Jiangzhe Zhu, Xiaowen Sun, Hao Cheng, Dawei Song, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  19. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547. ↗
  20. Md Tawkat Islam Khondaker, Numaan Naeem, Fatimah Lyba Khan, AbdelRahim A. Elmadany, and Muhammad Abdul-Mageed. 2024. Benchmarking AceGPT 13B GPT3.5 GPT4o ةيللادلا ةقلاعلا نيب ةطبارلا و "ريبعت" ةيرح" يه "ريبعتلا "ريبعت" نأ لعفلا ىلإ ريشي يتلا ةيلمعلا وأ اهب موقت .ريبعتلا ةيرحلا ةقلاعلا ةيللادلا نيب ةطبارلا و "ريبعت" ةيرح" يه "ريبعتلا "ريبعت" نأ ىلإ ريشي وأ لعفلا يتلا ةيلمعلا اهب موقت ةيرحلا .ريبعتلا سيِ ِ سْأَّتلَا م ه ُ س ْ َأ ةقلاعلا ةيللادلا نيب ةطبارلا و )ريِب ع ْ َت( ةَّي رُ ِ ح( يه ) ري ِ ِب ع ْ َّتلَا . مزلات ةقلاع يوغللا رذجلا ) مولم ( ةملكل . ) م و ل ( وه يوغللا رذجلا " مولم " ةملكل ". م و ل " وه رذجلا ةملكل يوغللا : وه ) مولم ( . م و ل find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rasheed-2025-evaluating-rag-pipelines-arabic-lexical/