Elamur Research⌕
☰
home / papers / 2026 / EncouRAGe: Evaluating RAG Local, Reliable, and Efficient

EncouRAGe: Evaluating RAG Local, Reliable, and Efficient

Paper · 2026 · Jan Strich, Martin Semmann and Chris Biemann

Summary

Comparing RAG pipelines on domain-specific data is difficult because current tools such as RAGAS, RAGChecker and TruLens mostly provide metrics. They lack standardized reference implementations of RAG methods, allow little customization of LLM-as-a-judge prompts and tend to assume cloud deployment. To fill these gaps, the authors present EncouRAGe, an open-source Python library composed of five modules, namely Metrics, Inference, Vector Store, the RAG Factory and the Type Manifest. Datasets are converted into typed question–answer–gold-context objects, and prompts are assembled with Jinja2 templates. The library runs local inference through vLLM using the OpenAI SDK and stores embeddings in Chroma or Qdrant. Its metrics cover the generator, the retriever and LLM-as-a-judge scoring, which helps show whether errors come from retrieval or from generation.

In a case study on four QA benchmarks (about 25k questions and roughly 50k documents) using Gemma3 and E5-Large, combining BM25 with dense retrieval gave the strongest overall results. Query rewriting and summarization methods added cost for little benefit. The tool is useful for reproducible and private RAG benchmarking.

Key points

  • Library contents: The RAG Factory groups methods into Without RAG (Pretrained-Only, Oracle Context), Basic (Vanilla, BM25, Hybrid BM25, Reranker) and Advanced (HyDE, Summarization, SumContext). The paper reports both 9 and 10 methods, and the library includes over 20 metrics.

  • Evaluation setup: The datasets are HotPotQA, FeTaQA, FinQA and BioSQA, run zero-shot with Gemma3 as the generator and E5-Large as the embedding model.

  • Hybrid BM25 F1 against Oracle Context:

    DatasetHybrid BM25Oracle Context
    HotPotQA40.843.4
    FeTaQA50.149.4
    FinQA51.872.9
    BioSQA49.954.5

    FeTaQA is the one dataset where Hybrid BM25 slightly exceeds the oracle.

  • Overstated claims: The table does not fully support two of the paper's statements.

    • Hybrid BM25 is described as best on every metric, yet it is not top on Recall@10 for HotPotQA (BM25 reaches 98.4) or FeTaQA (Vanilla RAG reaches 92.7).
    • The paper says RAG still trails the Oracle Context, which is not the case on FeTaQA.

  • Weaker methods: The reranker brought no MRR gain. HyDE helped only on BioSQA. Summarization raised retrieval scores but lowered F1. Pretrained-Only scored just 9.6 on FinQA, suggesting financial data is poorly covered in pretraining.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany79 works · 5,422 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Jan Strich, Martin Semmann and Chris Biemann. 2026. EncouRAGe: Evaluating RAG Local, Reliable, and Efficient. Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.rag4reports-1.5

@inproceedings{strich2026encourage,
  title = {{EncouRAGe: Evaluating RAG Local, Reliable, and Efficient}},
  author = {Jan Strich, Martin Semmann and Chris Biemann},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.rag4reports-1.5},
  url = {https://doi.org/10.18653/v1/2026.rag4reports-1.5}
}

References

46 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Confident AI. 2024. DeepEval: The Open-Source LLM Evaluation Framework. Https://deepeval.com. find
  2. Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. Complementing Lexical Retrieval with Semantic Residual Embedding. arXiv preprint. ArXiv:2004.13969. — [arXiv:2004.13969](https://arxiv.org/abs/2004.13969) find
  3. Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. Https://www.wandb.com/. find
  4. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Li…
  5. Yifu Chen, Shengpeng Ji, Haoxiao Wang, Ziqing Wang, Siyu Chen, Jinzheng He, Jin Xu, and Zhou Zhao. 2025a. WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
  6. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages…
  7. Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, YiQiu Guo, Yanfeng Wang, and Yu Wang. 2025b. Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti… find
  8. Yizheng Huang and Jimmy Huang. 2024. A Survey on Retrieval-Augmented Text Generation for Large Language Models. arXiv preprint. ArXiv:2404.10981. — [arXiv:2404.10981](https://arxiv.org/abs/2404.10981) find
  9. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Proceedings of the 2021 Conference on Empirical Methods in Natural…
  10. HuggingFace. 2025. Evaluate: A library for easily evaluating machine learning models and datasets. Https://github.com/huggingface/evaluate. find
  11. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422–446. find
  12. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, and et al. Mesnard. 2025. Gemma 3 Technical Report. arXiv preprint. ArXiv:2503.19786. find
  13. Chroma Team. 2025. Chroma: Open-source search and retrieval database for AI applications. Https://github.com/chroma-core/chroma. find
  14. Comet ML, Inc. 2025. Comet — The AI Developer Platform. Https://www.comet.com/site/. find
  15. Kiseung Kim and Jay-Yoon Lee. 2024. RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation. In Proceedings of the Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, and Li Yuan. 2024. find
  16. Chatlaw: A Multi-Agent Collaborative Legal 2024 Conference on Empirical Methods in Natural Language Processing, pages 22149–22161, Miami, Florida, USA. Association for Computational Linguistics. find
  17. Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022. FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics, 10:35–49.
  18. Anastasia Krithara, James G. Mork, Anastasios Nentidis, and Georgios Paliouras. 2023. The Road From Manual To Automatic Semantic Indexing Of Biomedical Literature: a 10 Years Journey. Frontiers in Research Metrics and Analytics, 8. find
  19. Irina Nikishina, Özge Sevgili, Mahei Manhai Li, Chris Biemann, and Martin Semmann. 2025. Creating a Taxonomy for Retrieval Augmented Generation Applications. arXiv preprint. ArXiv:2408.02854. — [arXiv:2408.02854](https://arxiv.org/abs/2408.02854) find
  20. OpenAI. 2025a. OpenAI Evals — A framework for evaluating LLMs. Https://github.com/openai/evals. find
  21. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pages 61…
  22. OpenAI. 2025b. openai-python: Official Python library for the OpenAI API. Https://github.com/openai/openai-python. find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  24. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Informatio…
  25. Qdrant Team. 2025. Qdrant: High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Https://github.com/qdrant/qdrant. find
  26. Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019. Multi-Interest Network with Dynamic Routing for Recommendation at Tmall. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIK… find
  27. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find
  28. Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. RAGChecker: A Fine-grained Framework for Diagnosing Retriev… find
  29. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  30. Jerry Liu. 2022. LlamaIndex. Https://github.com/jerryjliu/llama_index. find
  31. Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. Streamlining Evaluation with ir-measures. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part II, page 305–310, Berlin, Heidelberg. Springer-Verlag. find
  32. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  33. Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. 2007. GLEU: Automatic Evaluation of Sentence-Level Fluency. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344–351, Prague, Czech Republic. Association for Computational Linguistics. find
  34. Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11709–11724, Miami, Flo… find
  35. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie,… — RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
  36. Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Fen Xie, and Corey Zumar. 2018. Accelerating the Machine Learning Lifecycle with MLflow. IEEE Data Eng. Bull., 41:39–45. find
  37. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguisti…
  38. Snowflake Inc. 2024. Https://www.trulens.org. find
  39. Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437. find
  40. Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Chris Biemann, and Martin Semmann. 2025. T^2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation. Preprint, arXiv:2506.12071.
  41. Ellen M. Voorhees. 1993. Using WordNet to disambiguate word senses for text retrieval. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’93, pages 171–180, Pittsburgh, Pennsylvania, USA. Association for Computing Machine… find
  42. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report. arXiv preprint. ArXiv:2402.05672.
  43. Yuchen Wang, Shangxin Guo, and Chee Wei Tan. 2025. From Code Generation to Software Testing: AI Copilot With Context-Based Retrieval-Augmented Generation. IEEE Software, 42(4):34–42. find
  44. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking Retrieval-Augmented Generation for Medicine. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand. Association for Computational Linguistics.
  45. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2…
  46. Matei A. Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/strich-2026-encourage-evaluating-rag-local-reliable/