Elamur Research⌕
☰
home / papers / 2025 / RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

Paper · 2025 · Kunlun Zhu, Yifan Luo, Dingling Xu et al.

Summary

Testing RAG systems in specialised fields such as law, finance or medicine is hard. Building expert-annotated, domain-specific test sets is expensive and raises privacy issues, and lexical metrics like BLEU or ROUGE-L judge long answers poorly. RAGEval addresses this by generating evaluation data automatically. Starting from a handful of seed documents, an LLM drafts a domain schema of abstract key elements, which humans then refine. Rule-based sampling and LLMs fill the schema with concrete values ("configurations"), and GPT-4o turns these into synthetic documents. From the documents the pipeline derives question–answer pairs, supporting references and factual keypoints for each answer. Responses are scored against these keypoints using three metrics: Completeness, Hallucination and Irrelevance. The generated samples beat zero-shot and one-shot generation on clarity, safety, conformity and richness, and LLM-based scoring agrees closely with human judges. The authors release the DragonBall benchmark built with this pipeline. It offers a cheaper, more reproducible way to build domain-specific RAG evaluations.

Key points

  • The pipeline runs as schema → configuration → document → (Q, A) → references → keypoints. It covers scenarios such as finance, law and medicine in Chinese and English, and code and data are at github.com/OpenBMB/RAGEval.
  • In the retriever comparison, BM25 matched or beat dense models on generation metrics. BGE-M3 had slightly lower Recall (55.10% EN) but the best Completeness (64.08% EN), while GTE-multilingual-Base trailed (52.55% CN Recall).
  • Raising TopK from 2 to 8 lifted Recall from 49.18% to 78.94% (CN) and from 38.16% to 64.05% (EN). Completeness gains shrank after TopK=5.
  • The best Chunk-TopK setting depends on query type: 256-10 for factual questions, 512-5 for multi-hop reasoning and 128-20 for numerical comparison.
  • GPT-4o scored best overall, but leading open-source models were close behind. The main limitations are LLM hallucination during data generation and the cost of closed-source models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓Northeastern UniversityBoston · United States57 works · 6,533 citations🎓Beijing Normal UniversityBeijing · China15 works · 3,177 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu and Maosong Sun. 2025. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.418

@inproceedings{zhu2025rageval,
  title = {{RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework}},
  author = {Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu and Maosong Sun},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.418},
  url = {https://doi.org/10.18653/v1/2025.acl-long.418}
}

References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. CoRR, abs/2405.13576. find
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. 2024. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187.
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  5. Tilmann Bruckhaus. 2024. Rag does not work for enterprises. Preprint, arXiv:2406.04369. — [arXiv:2406.04369](https://arxiv.org/abs/2406.04369) find
  6. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2318–2335, Bang…
  7. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  8. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  9. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic…
  10. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  11. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  12. Jerry Liu. 2023. Building production-ready rag applications. find
  13. Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043. — [arXiv:2401.17043](https://arxiv.org/abs/2401.17043) find
  14. Isabelle Nguyen. 2023. Evaluating rag part i: How to evaluate document retrieval. find
  15. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2017. MS MARCO: A human-generated MAchine reading COmprehension dataset.
  16. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  18. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476. find
  19. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries.
  20. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  21. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. 2024. Crag–comprehensive rag benchmark. arXiv preprint arXiv:2406.04744. — [arXiv:2406.04744](https://arxiv.org/abs/2406.04744) find
  22. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–…
  23. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-rageval-scenario-specific-rag-evaluation/