Elamur Research⌕
☰
home / papers / 2024 / Exploring Retrieval Augmented Generation in Arabic

Exploring Retrieval Augmented Generation in Arabic

Summary

Retrieval-augmented generation has been studied mostly for English. It is unclear how well current embedding models and LLMs handle Arabic, which has rich morphology and many dialects that differ sharply from MSA. The authors build a complete Arabic RAG pipeline and evaluate it in two steps. First they compare multilingual semantic embedding models as retrievers. Then they connect the best retriever to several LLMs that act as generators.

They use two QA collections. ARCD is built from Wikipedia. Ar_EduText is a new small set made from high-school textbook passages, with GPT-generated questions and answers that were checked by hand. It also includes Egyptian-dialect rewrites of the questions, so the authors can test what happens when the query dialect differs from the document dialect. In addition, they try automatically rewriting context-dependent ARCD questions so each one stands alone.

Contrary to the authors' expectations, off-the-shelf models cope well. E5-large and BGE are the strongest retrievers. Among open LLMs, Llama3 and Mistral look promising, while GPT-3.5 Turbo leads on the textbook set. The study gives a practical baseline for RAG in a dialect-rich language with limited resources, and the code and data are public.

Key points

  • Ar_EduText was built from six MSA textbook passages and has 158 questions. GPT-4o drafted the questions and GPT-3.5 Turbo drafted the answers, and humans then kept, edited or rejected them. About 75.3% of the Egyptian-dialect question rewrites needed manual fixing.
  • ARCD contains 1,395 crowdsourced questions from 155 Wikipedia articles, covering 460 unique paragraphs. Many of its questions depend on an earlier question for context.
  • On Ar_EduText, the generators scored as follows (F1 / BLEU / cosine):
    • GPT-3.5 Turbo: 0.59 / 0.33 / 0.95
    • Llama3-70b: 0.43 / 0.20 / 0.91
    • Mixtral: 0.45 / 0.13 / 0.84
    • Mistral-7b: 0.26 / 0.08 / 0.89
    • quantized JAIS-7b: 0.37 / 0.17 / 0.72
  • On ARCD, Mistral-7b came first (F1 0.22, cosine 0.83), just ahead of Llama3-70b (0.18, 0.82), with Mixtral third (0.22, 0.79).
  • The open models wrapped their answers in extra text. Scoring them required post-processing written separately for each model, and the sample outputs suggest that better prompting could improve results.

Interesting ideas

  • I1. Make a dialect-shifted copy of every test query and retrieve against documents in the standard variety, so robustness to register change becomes a measurable number. — For Mova (learn): Rewrite evaluation questions into informal or regional English and use them as a control that the extracted frames and answers should not change.
  • I2. Rewrite a follow-up question into a self-contained one using only the previous question. Change it only when a vague reference makes it unclear, and never treat proper names as ambiguous. — For Mova (frames): A conservative rule like this could guide coreference over question sequences in fable QA, replacing pronouns with explicit entities before answering.
  • I3. Keep both the raw machine-generated item and its human-corrected version side by side, instead of overwriting the draft. — For Mova (learn): Storing teacher-tier outputs next to their corrections would show how often the teacher errs and would provide pairs for inducing repair rules.
  • I4. Evaluate in stages: lock in the best retriever first, then compare generators on top of it, so the two kinds of error do not mix. — For Mova (answer): Score the world-of-text construction separately from the answering reader, so a wrong answer can be traced to one stage.
  • I5. Score answers with several metrics at once (token F1, BLEU, embedding cosine), and extract the core answer from wordy outputs before scoring. — For Mova (back): Generated text from the graph → text path could be judged the same way, separating content correctness from surface wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.07425 · arXiv 2408.07425

@misc{elbeltagy2024exploring,
  title = {{Exploring Retrieval Augmented Generation in Arabic}},
  author = {El-Beltagy, Samhaa R. and Abdallah, Mohamed A.},
  year = {2024},
  doi = {10.48550/arxiv.2408.07425},
  eprint = {2408.07425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.07425}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gao, Yunfan, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. (2024). "Retrieval-Augmented Generation for Large Language Models: A Survey". https://arxiv.org/abs/2312.10997. ↗ our card
  2. Darwish, Kareem, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, et al. (2021). "A panoramic survey of natural language processing in the Arab world". Communications of the ACM 64. ACM New York, NY, USA: 72–81. find
  3. Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". In Advances in Neural Information Processing Systems, ed. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, 33:9459–9474. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf. ↗
  4. Huang, Yizheng, and Xiangji Huang. (2024). "A Survey on Retrieval-Augmented Text Generation for Large Language Models". ArXiv abs/2404.10981. https://api.semanticscholar.org/CorpusID:269188036. ↗
  5. Chirkova, Nadezhda, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. (2024). "Retrieval-augmented generation in multilingual settings". https://arxiv.org/abs/2407.01463. ↗
  6. Abdelazim, Hazem, Mohamed Tharwat, and Ammar Mohamed. (2023). "Semantic Embeddings for Arabic Retrieval Augmented Generation (ARAG).". International Journal of Advanced Computer Science & Applications 14 (11): 1328 –1333 . find
  7. Mozannar, Hussein, Karl El Hajal, Elie Maamary, and Hazem Hajj. (2019). "Neural Arabic question answering". arXiv preprint arXiv:1906.05394. ↗
  8. Neelakantan, Arvind, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, et al. (2022). "Text and Code Embeddings by Contrastive Pre-Training". https://arxiv.org/abs/2201.10005. ↗ our card
  9. Alsuhaibani, Mohammed. (2023). "Deep Learning-based Sentence Embeddings using BERT for Textual Entailment". International Journal of Advanced Computer Science and Applications 14. Science and Information (SAI) Organization Limited. find
  10. Kayid, Amr, and Nils Reimers. (2022). "Cohere’s Multilingual Text Understanding Model is Now Available". https://cohere.com/blog/multilingual. December 12. ↗
  11. Reimers, Nils, and Iryna Gurevych. (2019). "Sentence-bert: Sentence embeddings using siamese bert-networks". arXiv preprint arXiv:1908.10084. ↗
  12. Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter". arXiv preprint arXiv:1910.01108. ↗
  13. Duquenne, Paul-Ambroise, Holger Schwenk, and Beno\^ıt Sagot. (2023). "SONAR: sentence-level multimodal and language-agnostic representations". arXiv e-prints: arXiv–2308. find
  14. Wang, Liang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. (2022). "Text Embeddings by Weakly-Supervised Contrastive Pre-training. CoRR abs/2212.03533 (2022)". ↗ our card
  15. Dumais, Susan T, George W Furnas, Thomas K Landauer, Scott Deerwester, and Richard Harshman. (1988). "Using latent semantic analysis to improve access to textual information". In Proceedings of the SIGCHI conference on Human factors in computing systems, 281–285. find
  16. Deerwester, Scott, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. (1990). "Indexing by latent semantic analysis". Journal of the American society for information science 41. Wiley Online Library: 391–407. find
  17. Mikolov, Tomas, Greg Corrado, Kai Chen, and Jeffrey Dean. (2013). "word2vec-v1". Proceedings of the International Conference on Learning Representations (ICLR 2013): 1–12. find
  18. Peters, Matthew E, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. (2018). "Deep Contextualized Word Representations". In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), ed. Marilyn Walker, Heng Ji, and Amanda Stent, 2227–2237. New Orleans, Louisiana: Association for Computational Linguistics. https://aclanthology.org/N18-1202. ↗
  19. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. (2018). "Bert: Pre-training of deep bidirectional transformers for language understanding". arXiv preprint arXiv:1810.04805. ↗
  20. Soliman, Abu Bakr, Kareem Eissa, and Samhaa R. El-Beltagy. (2017). "AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP". In Procedia Computer Science. find
  21. Antoun, Wissam, Fady Baly, and Hazem Hajj. (2020). "Arabert: Transformer-based model for arabic language understanding". arXiv preprint arXiv:2003.00104. ↗ our card
  22. Ollama. (2024). "Ollama Embedding models". https://ollama.com/blog/embedding-models. April. ↗
  23. Sengupta, Neha, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, et al. (2023). "Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models". arXiv preprint arXiv:2308.16149. ↗ our card
  24. Xiao, Shitao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. (2023). "C-Pack: Packaged Resources To Advance General Chinese Embedding". find
  25. Jiang, Albert Q, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, et al. (2023). "Mistral 7B". https://arxiv.org/abs/2310.06825. ↗
  26. Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, et al. (2024). "The Llama 3 Herd of Models". https://arxiv.org/abs/2407.21783. ↗
  27. Jiang, Albert Q, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, et al. (2024). "Mixtral of Experts". https://arxiv.org/abs/2401.04088. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/el-2024-beltagy-exploring-retrieval-augmented-generation-arabic/