Elamur Research⌕
☰
home / papers / 2025 / OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

Paper · 2025 · Shuting Wang, Jiejun Tan, Zhicheng Dou et al.

Summary

Existing RAG benchmarks tend to cover few scenarios and judge systems along few dimensions. This makes it hard to see how retrieval-augmented systems behave in specialised fields such as finance. OmniEval addresses this with a Chinese-language financial benchmark built on a two-axis grid that crosses five task types with 16 finance topics. The authors assembled a retrieval corpus from public financial challenge data, finance-related Wikipedia pages, a pretraining corpus and crawled official agency sites. They then used a chained pipeline in which GPT-4 classifies topics, generates question–answer instances and screens their quality, with human reviewers correcting a subset.

Evaluation covers both stages of the pipeline:

  • Retrieval is scored with rule-based metrics such as MAP.
  • Generation is scored with Rouge-style metrics plus learned judges that check aspects like hallucination and numerical correctness.

Experiments pair several retrievers with several large open LLMs. Scores vary noticeably across topic–task cells, and there is considerable headroom overall. The grid view helps locate specific weaknesses of a RAG system rather than hiding them in one average.

Key points

  • Taxonomy: five tasks (extractive, multi-hop reasoning, contrast, long-form, conversational QA) crossed with 16 financial subtopics, such as Fund, Stock Market and AML.
  • Data:
    • 11.4k auto-generated test items;
    • 1.7k human-annotated test items;
    • 3k auto-generated training items.
  • Corpus: drawn from BSCF, FinGLM, Wiki-Fin, BAAI-Fin and official web pages. It was chunked via LlamaIndex into 2048-length passages with 256 overlap.
  • Judge model: a fine-tuned Qwen2.5-7B-Instruct reached 74.4% accuracy, outperforming prompted LLM judges.
  • Systems tested:
    • Retrievers: BGE-M3, BGE-large-zh, GTE-Qwen2-1.5b and jina-zh.
    • Generators: Qwen2.5-72B-Instruct, Llama3.1-70B-Instruct, Deepseek-v2-chat and Yi15-34B.
    • Code is released at github.com/RUC-NLPIR/OmniEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Shuting Wang, Jiejun Tan, Zhicheng Dou and Ji-Rong Wen. 2025. OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.292

@inproceedings{wang2025omnieval,
  title = {{OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain}},
  author = {Shuting Wang, Jiejun Tan, Zhicheng Dou and Ji-Rong Wen},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.292},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.292}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024a. Fintextqa: A dataset for long-form financial question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangko… find
  2. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. CoRR, abs/2402.03216.
  3. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024c. Benchmarking large language models in retrieval-augmented generation. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth…
  4. Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, and Zhongyu Wei. 2023. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning. CoRR, abs/2310.15205. — [arXiv:2310.15205](https://arxiv.org/abs/2310.15205) find
  5. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. Finqa: A dataset of numerical reasoning over financial data. Proceedings of EMNLP 2021.
  6. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. Proceedings of EMNLP 2022.
  7. DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez,… — The Llama 3 Herd of Models
  9. Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. CoRR, abs/23…
  10. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. CoRR, abs/2311.11944. — [arXiv:2311.11944](https://arxiv.org/abs/2311.11944) find
  11. Xiang Li, Zhenyu Li, Chen Shi, Yong Xu, Qing Du, Mingkui Tan, and Jun Huang. 2024. Alphafin: Benchmarking financial analysis with retrieval-augmented stock-chain framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC… find
  12. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  13. Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. RECALL: A benchmark for llms robustness against external counterfactual knowledge. CoRR, abs/2311.08147. — [arXiv:2311.08147](https://arxiv.org/abs/2311.08147) find
  14. Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. CRUD-RAG: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. CoRR, abs/2401.17043. — [arXiv:2401.17043](https://arxiv.org/abs/2401.17043) find
  15. Isabelle Mohr, Markus Krimmel, Saba Sturua, Mohammad Kalim Akram, Andreas Koukounas, Michael Günther, Georgios Mastrapas, Vinit Ravishankar, Joan Fontanals Martínez, Feng Wang, et al. 2024. Multi-task contrastive learning for 8192-token bilingual text embeddings. arXiv preprint arXiv:2402.17016. — [arXiv:2402.17016](https://arxiv.org/abs/2402.17016) find
  16. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  17. CoRR, Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: an automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find
  18. Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 84–90, Parramatta, Australia. find
  19. Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. WHEN FLUE MEETS FLANG: benchmarks and large pre-trained language model for financial domain. CoRR, abs/2211.00083. find
  20. Ankur Sinha and Tanmay Khandait. 2020. Impact of news on the commodity market: Dataset and results. CoRR, abs/2009.04202. — [arXiv:2009.04202](https://arxiv.org/abs/2009.04202) find
  21. Yejun Soun, Jaemin Yoo, Minyong Cho, Jihyeong Jeon, and U Kang. 2022. Accurate stock movement prediction with self-supervised learning from sparse noisy tweets. In 2022 IEEE International Conference on Big Data (Big Data), pages 1691–1700. IEEE Computer Society. find
  22. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin…
  23. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  24. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  25. Shuting Wang, Jiongnan Liu, Shiren Song, Jiehan Cheng, Yuqi Fu, Peidong Guo, Kun Fang, Yutao Zhu, and Zhicheng Dou. 2024a. Domainrag: A chinese Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023b. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.…
  26. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Ya… — [arXiv:2402.12659](https://arxiv.org/abs/2402.12659) find
  27. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. PIXIU: A large language model, instruction data and evaluation benchmark for finance. CoRR, abs/2306.05443. — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) find
  28. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 6233–6251. Association for Computational Li…
  29. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue… — Yi: Open Foundation Models by 01.AI
  30. Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, and Jianfeng Gao. 2024. Reeval: Automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 1… find
  31. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023a. C-pack: Packaged resources to advance general chinese embedding. CoRR, abs/2309.07597.
  32. Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. 2024b. Richrag: Crafting rich responses for multi-faceted queries in retrieval-augmented generation. CoRR, abs/2406.12566. *Figure 8: Rouge-L scores of generators on topic-specific auto-generated subsets.* benchmark for eval…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-omnieval-omnidirectional-automatic-rag-evaluation/