Elamur Research⌕
☰
home / papers / 2026 / T²-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation

T²-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation

Paper · 2026 · Jan Strich, Enes Kutay Isgorur, Maximilian Trescher et al.

Summary

RAG systems are usually tested on plain text or Wikipedia-style data. Existing text-and-table QA sets assume the relevant passage is already given (the Oracle Context setting). Their questions also often only make sense alongside that passage, so several answers could be valid across a corpus. T$^2$-RAGBench fixes this by turning three financial QA datasets (FinQA, ConvFinQA, TAT-DQA) into a retrieval benchmark. Llama 3.3-70B rewrote each question so it has one unambiguous answer without seeing the context. The result is 23,088 question-context-answer triples over 7,318 real financial documents, where each context joins report text with a markdown table. The authors define a tolerant numeric metric, Number Match, which accepts small rounding differences and unit-scale shifts, and use MRR@k and R@k for retrieval. They compare many RAG methods and embedding models and study how corpus size affects retrieval. Hybrid BM25, which merges dense and sparse retrieval, performs best. Even so, it trails the Oracle Context setting by almost 30% in Number Match. This shows that retrieving mixed text-and-table content is still an open problem.

Key points

  • The benchmark has three subsets: FinQA (8,281 QA pairs), ConvFinQA (3,458, first-turn questions only) and TAT-DQA (11,349). They come from 38,954 original pairs, and FinQA and ConvFinQA draw on FinTabNet documents.
  • Experts judged 91.3% of the reformulated questions to be context-independent, checking 100 samples per subset with four annotators.
  • Number Match scores numeric answers with a tolerance threshold and corrects for order-of-magnitude (unit) differences.
  • Hybrid BM25 is the strongest retriever, and current SOTA embedding models reach low R@5 and MRR@5 on text-and-table contexts.
  • Most errors come from arithmetic slips, parsing problems and unnecessary reasoning, while about 6% trace back to faulty question reformulation. Limitations include finance-only data and the use of quantized models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany79 works · 5,422 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Chris Biemann and Martin Semmann. 2026. T²-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.8

@inproceedings{strich2026ragbench,
  title = {{T²-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation}},
  author = {Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Chris Biemann and Martin Semmann},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.8},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.8}
}

References

47 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations, Vienna, Austria. ICLR.
  2. Dipali Baviskar, Swati Ahirrao, Vidyasagar Potdar, and Ketan V. Kotecha. 2021. Efficient Automated Processing of the Unstructured Documents Using Artificial Intelligence: A Systematic Literature Review and Future Directions. IEEE Access, 9:72894–72936. find
  3. Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. FinTextQA: A Dataset for Long-form Financial Question Answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6025–6047,… find
  4. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data. In Findings of the Association for Computational Linguistics, volume EMNLP 2020 of Findings of ACL, pages 1026–1036, Online Eve…
  5. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Proceedings of the 2021 Conference on Empirical Methods in Natural…
  6. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, p…
  7. Chanyeol Choi, Jihoon Kwon, Jaeseon Ha, Hojun Choi, Chaewoon Kim, Yongjae Lee, Jy-yong Sohn, and Alejandro Lopez-Lira. 2025. FinDER: Financial Dataset For Question Answering And Evaluating Retrieval-Augmented Generation. In Proceedings Of The 6th ACM International Conference On AI In Finance, pages… find
  8. Chroma Team. 2025. Chroma: Open-Source Search And Retrieval Database For AI Applications. Https://github.com/chroma-core/chroma. find
  9. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
  10. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023a. Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational L…
  11. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023b. Retrieval-Augmented Generation for Large Language Models: A Survey. Preprint, arXiv:2312.10997.
  12. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages…
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  14. Yulong Hui, YAO LU, and Huanchen Zhang. 2024. UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis. In Advances in Neural Information Processing Systems, volume 37, pages 67200–67217, Vancouver, BC, Canada. Curran Associates, Inc. find
  15. Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active Retrieval Augmented Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association…
  16. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. A Comprehensive Evaluation of Quantization Strategies for Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12186–12215, Bangkok, Thailand. Association for Com… find
  17. Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. 2025. MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 32–56, Albuquerque, New Mexico, USA. Association… find
  18. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611, Vancouver, Canada. Association…
  19. Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. Complement Lexical Retrieval Model With Semantic Residual Embeddings. In European Conference on Information Retrieval, pages 146–160, Virtual Event. Springer. find
  20. Pankaj Joshi, Aditya Gupta, Pankaj Kumar, and Manas Sisodia. 2024. find
  21. Robust Multi Model RAG Pipeline For Documents Containing Text, Table & Images. In 2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 993–999, Salem, India. IEEE. find
  22. Yannis Katsis, Saneem A. Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael R. Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2022. AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry. In Proceedings o… find
  23. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-Search-Predict: Composing Retrieval and Language Models For Knowledge-Intensive NLP. Preprint, arXiv:2212.14024.
  24. Kihun Kim, Mintae Kim, Hokyung Lee, Seong Ik Park, Youngsub Han, and Byoung-Ki Jeon. 2024. THoRR: Complex Table Retrieval and Refinement for RAG. In Proceedings of the Workshop Information Retrieval’s Role in RAG Systems (IR-RAG 2024) Co-Located with the 47th International ACM SIGIR Conference on Re… find
  25. Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023. Copy is All You Need. In The Eleventh International Conference on Learning Representations, Kigali, Rwanda. find
  26. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Informatio…
  27. Xinze Li, Yixin Cao, Yubo Ma, and Aixin Sun. 2024. Long Context vs. RAG for LLMs: An Evaluation and Revisits. Preprint, arXiv:2501.01880. — [arXiv:2501.01880](https://arxiv.org/abs/2501.01880) find
  28. Man Luo, Sharad Saxena, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. BioTABQA: Instruction Learning for Biomedical Table Question Answering. In Proceedings of the Working Notes of CLEF 2022 -Conference and Labs of the Evaluation Forum, volume 3180 of CEUR Workshop Proceedings, pages 291–304… find
  29. Linyong Nan, Chia-Hsuan Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryscinski, Nick Schoelkopf, Riley Kong, Xiangru Tang, Murori Mutuma, Benjamin Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir R. Radev. 2021. FeTaQA: Free-form Table…
  30. Irina Nikishina, Özge Sevgili, Mahei Manhai Li, Chris Biemann, and Martin Semmann. 2025. Creating a Taxonomy for Retrieval Augmented Generation Applications. Preprint, arXiv:2408.02854. — [arXiv:2408.02854](https://arxiv.org/abs/2408.02854) find
  31. Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural La…
  32. Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Vancouver, BC, Canada. find
  33. Sachin Raja, Ajoy Mondal, and C. V. Jawahar. 2023. find
  34. IC-DAR 2023 Competition on Visual Question Answering on Business Document Images. In Document Analysis and Recognition, pages 454–470, Cham, Germany. Springer Nature Switzerland. find
  35. Carl M Rebman Jr, Queen E Booker, Hayden Wimmer, Steve Levkoff, Mark McMurtrey, and Loreen Marie Powell. 2023. An Industry Survey of Analytics Spreadsheet Tools Adoption: Microsoft Excel vs Google Sheets. Information Systems Education Journal, 21(5):29–42. find
  36. Sohini Roychowdhury, Marko Krema, Anvar Mahammad, Brian Moore, Arijit Mukherjee, and Punit Prakashchandra. 2024. ERATTA: Extreme RAG for Table To Answers with Large Language Models. Preprint, arXiv:2405.03963. — [arXiv:2405.03963](https://arxiv.org/abs/2405.03963) find
  37. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. In The Twelfth International Conference on Learning Representations, Vienna, Austria. The Association for Computational Lingu…
  38. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2021. Document Collection Visual Question Answering. In 16th International Conference on Document Analysis and Recognition, volume 12822 of Lecture Notes in Computer Science, pages 778–792, Lausanne, Switzerland. Springer. find
  39. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Multilingual E5 Text Embeddings: A Technical Report. Preprint, arXiv:2402.05672.
  40. Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. 2024b. Beyond The Limits: A Survey Of Techniques To Extend The Context Length In Large Language Models. In Proceedings Of The Thirty-Third International Joint Conference On Artificial Intelligence, Jeju,…
  41. Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Tongliang Li, Zhoujun Li, and Guanglin Niu. 2025. TableBench: A Comprehensive and Complex Benchmark for Table Question Answering. In Association for the Advancement of Artificial… find
  42. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective Retrieval Augmented Generation. Preprint, arXiv:2401.15884.
  43. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2…
  44. Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. 2021. Global Table Extractor (GTE): A Framework For Joint Rable Identification and Cell Structure Recognition Using Visual Context. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 697–706, Los… find
  45. Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022. Towards Complex Document Understanding By Discrete Reasoning. In MM ’22: The 30th ACM International Conference on Multimedia, pages 4857–4866, Lisboa, Portugal. ACM. find
  46. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics…
  47. Fengbin Zhu, Junfeng Li, Liangming Pan, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat-Seng Chua. 2025. Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Finance. In Proceedings of the 33rd ACM International Conference on Multimedia, Mm ’25, pages 6289–6297, Dublin, Irela… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/strich-2026-t-2-ragbench-text-table/