Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese
Paper · 2026 · José Matos, Catarina Silva and Hugo Gonçalo Oliveira
Summary
The paper asks whether retrieval-augmented generation helps small open language models answer scientific multiple-choice questions in European Portuguese (pt-PT), and how much the outcome depends on retrieval design. Earlier Portuguese RAG studies focused mainly on the Brazilian variety. The authors evaluate five instruction-tuned models (4–12B) using the Portuguese portion of the Global MMLU Lite test data. External evidence comes from a vector index built from more than 32,000 Portuguese scientific documents in the CorEGe-PT corpus. Each model runs closed-book and in 16 retrieval setups, which vary the dense encoder (multilingual or Portuguese-specific), the use of a reranker, and the number of retrieved chunks.
The benefit of retrieval varies strongly by model:
- Gemma3: large and consistent gains.
- AMALIA and EuroLLM: small gains that depend on the configuration.
- Qwen3-8B: the strongest closed-book model, but slightly worse with retrieval, especially with longer contexts.
The study gives a reproducible pt-PT baseline. It shows that retrieval settings need tuning per model and that retriever and reranker language and domain should match the task.
Key points
- Setup:
- Benchmark: Global MMLU Lite PT split (400 professionally translated items).
- Models: AMALIA, EuroLLM-9B-Instruct, Qwen3-8B, Gemma3-4B-IT, Gemma3-12B-IT.
- Evaluation: 3-shot prompting with log-likelihood scoring in lm_evaluation_harness.
- Retrieval pipeline:
- Chunking: section-based splits, then 500-character chunks with 50-character overlap, stored in ChromaDB.
- Encoders: paraphrase-multilingual-mpnet-base-v2 vs. serafim-335m-portuguese-pt-sentence-encoder-ir.
- Reranker: gte-multilingual-reranker-base.
- Context size: k ∈ {1, 3, 5, 10}.
- Largest significant gains (p < 0.05): Gemma-3-12B-IT went from 55.5% to 71.0% (+15.5 pp), and Gemma-3-4B-IT from 50.7% to 60.5% (+9.8 pp).
- Small or negative effects: AMALIA improved by at most +2.7 pp and EuroLLM by at most +2.2 pp, with mean gains near zero. Qwen3-8B (73.0% closed-book) lost 2.89 pp on average.
- Reranking: it did not reliably raise accuracy, but it tended to move the best k toward smaller contexts.
Institutions
PortugalFields, topics and concepts
📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works
🧭 Topics 2Research subjects and languages the work is about.Question answering · 441 worksInformation retrieval & search · 473 works
Cite
Cite as José Matos, Catarina Silva and Hugo Gonçalo Oliveira. 2026. Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese. Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026.
@inproceedings{matos2026benchmarking,
title = {{Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese}},
author = {José Matos, Catarina Silva and Hugo Gonçalo Oliveira},
booktitle = {Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026},
year = {2026},
doi = {10.63317/3muergicuxwk},
url = {https://doi.org/10.63317/3muergicuxwk}
}References
- Leandro Costa and João Baptista de Oliveira e Souza-Filho. 2024. Adapting LLMs to New Domains: A Comparative Study of Fine-Tuning and RAG strategies for Portuguese QA Tasks. In Proceedings of the 15th Brazilian Symposium in Information and Human Language Technology, pages 217–227, Belém do Pará, Bra… find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/matos-2026-benchmarking-retrieval-augmented-generation-scientific/