📄 When Retrieval Hurts: Evidence Utilization, Script Fidelity, and Knowledge Conflicts in Multilingual RAG

Summary

The paper asks why retrieval-augmented generation (RAG) often fails to improve, and sometimes worsens, extractive question answering across many languages. The authors compare RAG with a non-RAG baseline over 21 typologically diverse languages in 9 writing scripts, using TyDiQA and XQuAD. They test five instruction-tuned LLMs (Gemma-2-9B-IT, Qwen2.5-7B-Instruct, LLaMA-2-7B-Chat, Aya-23-8B, EuroLLM-9B-Instruct) with five prompting strategies. The retrieval pipeline uses BGE-M3 dense retrieval with FAISS and sentence-aware chunking tuned to each language. The central finding is that retrieval is rarely the weak link. Relevant evidence is almost always retrieved, yet three of the five models score lower with RAG, because they fail to make use of what they receive. Models with stronger built-in knowledge suffer most from this interference. The authors also propose lightweight metrics for checking whether answers use the correct writing system. These metrics expose script errors that EM and F1 miss. The work suggests that multilingual RAG needs better grounding on the generator side and script-aware handling of conflicts, not just better retrievers.

Key points

• Data: 3,930 evaluation samples in total.
  • TyDiQA contributes 1,730 samples across 9 languages. Its secondary task (gold passage supplied) serves as No-RAG, and its primary task (full-corpus retrieval) serves as RAG.
  • XQuAD contributes 2,200 samples, evaluated without retrieval.
• Evidence-utilization gap: the retrieval hit rate (RHR) is 0.927, yet RAG lowers performance for three of five models. Gemma and Qwen are the most affected.
• New script-fidelity diagnostics: the authors introduce three measures, namely Transfer Rate (TR), Script Consistency (SC) and Language Match Rate (LMR).
  • Qwen produced Latin characters in Russian answers (LMR 0.565). Adding a Cyrillic-enforcing clause to the prompt raised LMR to 0.984 without retraining.
  • Korean Latin leakage (LMR 0.148) was also detected.
• Language-specific failures:
  • Telugu is the hardest language under RAG (F1 0.162–0.357), which the authors attribute to abugida tokenization fragmenting retrieved chunks.
  • In Arabic, negated facts are frequently turned into affirmatives (conflict rate about 0.50–0.60).
  • For Indonesian, minimal prompting reduced hallucination.
• XQuAD without retrieval: Aya-23-8B performs best (macro F1 0.757). This suggests that internal multilingual knowledge and the ability to exploit retrieved context are separate skills.