Elamur Research⌕
☰
home / papers / 2024 / Evaluation of Semantic Search and its Role in Retrieved-Augmented-Generation (RAG) for Arabic Language

Evaluation of Semantic Search and its Role in Retrieved-Augmented-Generation (RAG) for Arabic Language

Summary

The paper targets a gap in Arabic NLP: there is no standard benchmark for semantic search, and little is known about how the choice of retriever affects Arabic retrieval-augmented generation (RAG). The authors build two LLM-assisted evaluation sets. The first is a ranking benchmark of 2030 Arabic customer-support call summaries and 406 short queries, with graded relevance labels produced by GPT-4 and spot-checked by hand. The second is a RAG test made from 816 FAQ questions across four domains, each reworded three times by GPT-4. Five multilingual sentence encoders are scored with nDCG, MRR and mAP at cutoff 3, and compared against random and worst-case orderings. The same encoders are then plugged into a RAG loop: GPT-3.5-turbo writes the answers and GPT-4-turbo grades them. paraphrase-multilingual-mpnet-base-v2 wins on both tasks, but the other encoders do not keep the same order across the two settings. The authors explain this by the difference between asymmetric and symmetric search. The work gives Arabic practitioners a cheap, reproducible way to pick encoders for retrieval pipelines.

Key points

  • Ranking benchmark. GPT-4 wrote one query for each group of five summaries and labelled every pair as 0, 1 or 2 (irrelevant, somewhat relevant, very relevant).
    • A manual audit of 10% of the data (205 summaries, 41 queries) found only 2 confusions between the "very relevant" and "irrelevant" labels.
  • Encoders tested. The five encoders include Paraphrase Multilingual MiniLM (384-dim), Cmlm Multilingual (768-dim), paraphrase-multilingual-mpnet-base-v2 and sn-xlm-roberta-base-snli-mnli-anli-xnli.
    • Retrieval ranks documents by cosine similarity of the embeddings.
  • Ranking results. The best encoder (#3) reached nDCG@3 0.879, MRR@3 0.911 and mAP@3 0.888.
    • The random baseline, averaged over 30 shuffles, scored 0.669, 0.623 and 0.703.
    • The worst possible ordering scored 0.32, 0.138 and 0.401.
  • RAG setup. For each reworded query, the system retrieves the three closest stored questions in the same domain and passes their answers to GPT-3.5-turbo. GPT-4-turbo then checks the answer against the ground truth.
  • RAG results. Encoder #3 scored 63.11% with top-3 context and 63.84% with top-1, against 6.62% for random retrieval.
    • Top-1 context was slightly better than top-3, which the authors attribute to the generator blending the three retrieved answers.

Interesting ideas

  • I1. Bracketing each system's score between a random-ranking floor and a deliberately worst-ranking floor shows how much of a metric's range is real signal, rather than reporting raw numbers alone. — For Mova (learn): Evaluation reports for the parser or the readers could always include a shuffled baseline and an adversarial worst baseline as negative controls, so a gain can be judged against the metric's actual spread.
  • I2. Graded labels from an LLM teacher are made usable by a small manual audit that counts only gross errors (top label confused with bottom label), which is a cheap way to certify silver data. — For Mova (learn): LLM-teacher annotations of frames or arcs could be audited on a 10% sample that tracks only severe flips, and that rate could gate whether a silver batch enters training.
  • I3. The finding that an encoder's rank flips between long-document/short-query matching and length-matched matching warns that a component must be evaluated in the same configuration it will be used in. — For Mova (answer): Retrieval of supporting sentences for fable QA should be benchmarked separately for question-to-question matching and question-to-passage matching, since one score will not predict the other.
  • I4. Supplying fewer retrieved items to the generator sometimes helped, because extra near-miss context got merged into a wrong answer. — For Mova (answer): Readers could answer from the single best-supported frame and fall back to others only when it fails, instead of pooling all candidate evidence.
  • I5. Each test question was paraphrased three times by an LLM to test robustness to rewording rather than to exact surface form. — For Mova (tree): Generating several rewordings of treebank or test sentences and checking that the frames stay the same would expose parser decisions that depend on surface phrasing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.18350 · arXiv 2403.18350

@misc{mahboub2024evaluation,
  title = {{Evaluation of Semantic Search and its Role in Retrieved-Augmented-Generation (RAG) for Arabic Language}},
  author = {Mahboub, Ali and Za'ter, Muhy Eddin and Al-Rfooh, Bashar and Estaitia, Yazan and Jaljuli, Adnan and Hakouz, Asma},
  year = {2024},
  doi = {10.48550/arxiv.2403.18350},
  eprint = {2403.18350},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.18350}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333–2338, 2013. find
  2. Chetana Gavankar, Taniya Bhosale, Dhanashree Gunda, Anindita Chavan, and Shafa Hassan. A comparative study of semantic search systems. In 2020 International Conference on Computer Communication and Informatics (ICCCI), pages 1–7. IEEE, 2020. find
  3. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. ↗ our card
  4. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. find
  5. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Abdel Monem. Challenges in arabic natural language processing. In Computational linguistics, speech and image processing for arabic language, pages 59–83. World Scientific, 2019. find
  6. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507, 2021. find our card
  7. Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50–57, 1999. find
  8. Chung-Hong Lee and Hsin-Chang Yang. A classifier-based text mining approach for evaluating semantic relatedness using support vector machines. In International Conference on Information Technology: Coding and Computing (ITCC’05)-Volume II, volume 1, pages 128–133. IEEE, 2005. find
  9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. ↗
  10. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. find
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. ↗
  13. Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering, 32(8):1475–1488, 2019. find
  14. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024. ↗ our card
  15. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. Deep learning for arabic nlp: A survey. Journal of computational science, 26:522–531, 2018. find
  16. Ons Meddeb, Mohsen Maraoui, and Mounir Zrigui. Deep learning based semantic approach for arabic textual documents recommendation. In 2021 International Conference on INnovations in Intelligent SysTems and Applications (INISTA), pages 1–6. IEEE, 2021. find
  17. Aya M Al-Zoghby and Khaled Shaalan. Semantic search for arabic. In The Twenty-Eighth International Flairs Conference, 2015. find
  18. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331, 2023. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahboub-2024-evaluation-semantic-search-role-retrieved/