Elamur Research⌕
☰
home / papers / 2025 / TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering

TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering

Paper · 2025 · Junnan Zhu, Jingyi Wang, Bohan Yu et al.

Summary

TableEval addresses weaknesses in existing table question answering (TableQA) benchmarks. Most rely on simple flat tables, cover a single language, and may overlap with LLM pretraining data. The authors gathered tables from documents published in 2024, drawn from government, finance, academic and industry-report sources. The tables are written in Simplified Chinese, Traditional Chinese and English and include concise, hierarchical and nested layouts as well as flat ones. All questions are posed in Simplified Chinese. Question–answer pairs were produced through LLM-assisted generation (template- and role-based prompting), clustering-based deduplication, cross-model consistency checks and several rounds of human review.

The authors also introduce SEAT, an LLM-based judging framework. It extracts the answer to each sub-question, compares it with a structured reference and outputs JSON scores. SEAT agrees with human ratings more closely than exact-match, F1 or n-gram metrics.

Experiments show that closed-source models lead overall, although strong open reasoning models come close. All systems lose noticeable accuracy on complex layouts, Chinese tables and some specialised domains. This makes the benchmark a realistic test of structure-aware, multilingual table reasoning.

Key points

  • Coverage: TableEval spans 6 high-level tasks and 16 fine-grained question types, including lookup, numerical calculation, comparison, multi-hop reasoning and multi-turn dialogue. Every answer is objectively verifiable.
  • Leakage control: Source documents date from 2024, which reduces overlap with the pretraining corpora of most current LLMs.
  • Structural difficulty: Accuracy falls by roughly 10–15% on nested and hierarchical tables compared with flat ones.
  • Model comparison: o1-preview and Claude 3.5 Sonnet are generally the strongest models. DeepSeek-R1 and QwQ-32B-Preview are competitive, and DeepSeek-R1 reaches 90.13 on government-domain tables.
  • Availability and limitations: The dataset is released at github.com/wenge-research/TableEval. Image-based and highly irregular tables are not covered, and all questions are monolingual.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksQuestion answering · 441 works

Cite

Cite as Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang and Nan Xu. 2025. TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.363

@inproceedings{zhu2025tableeval,
  title = {{TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering}},
  author = {Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang and Nan Xu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.363},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.363}
}

References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, et al. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335. Association for Computational Li…
  2. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021a. Open question answering over tables and text. In International Conference on Learning Representations (ICLR), pages 1–18. find
  3. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations (ICLR), pages 1–25.
  4. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, et al. 2021b. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3697–3711. Association for Computational Linguistics.
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  6. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. find
  7. Naihao Deng, Zhenjie Sun, Ruiqi He, et al. 2024. Tables as texts or images: Evaluating the table reasoning ability of LLMs and MLLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 407–426. Association for Computational Linguistics. find
  8. Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. 2021. MATE: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7606– 7619. Association for Computational Linguistics. find
  9. Team GLM, :, Aohan Zeng, Bin Xu, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  11. Jiaqi Guo, Ziliang Si, Yu Wang, Qian Liu, Ming Fan, Jian-Guang Lou, Zijiang Yang, and Ting Liu. 2021. Chase: A large-scale and pragmatic Chinese dataset for cross-database context-dependent text-to-SQL. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and th… find
  12. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, et al. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4320–4333. Association for Computational Linguistics.
  13. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, et al. 2024. Qwen2.5-coder technical report. Preprint, arXiv:2409.12186.
  14. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1821–1831. Association for Computational Linguistics.
  15. Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. 2022. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  16. Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, et al. 2022. AIT-QA: Question answering dataset over complex tables in the airline industry. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find
  17. Vishwajeet Kumar, Yash Gupta, Saneem Chemmengath, Jaydeep Sen, Soumen Chakrabarti, Samarth Bharadwaj, and Feifei Pan. 2023. Multi-row, multi-span distant supervision for Table+Text question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), p… find
  18. Larissa R. Lautert, Marcelo M. Scheidt, and Carina F. Dorneles. 2013. Web table taxonomy and formalization. ACM SIGMOD Record, 42(3):28–33. find
  19. Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. 2023. S3HQA: A three-stage approach for multi-hop text-table hybrid question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1731–1740. Association for… find
  20. Jinyang Li, Binyuan Hui, Jiaxi Yang, Ge Qu, and others. 2023. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), pages 1–28. Curran A…
  21. Peng Li, Yeye He, and others. 2024. Table-gpt: Table fine-tuned gpt for diverse table tasks. In Proceedings of the ACM on Management of Data, pages 1–28. Association for Computing Machinery. find
  22. Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations (ICLR), pages 1–19.
  23. Linyong Nan, Chiachun Hsieh, Ziming Mao, et al. 2022. FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35–49.
  24. OpenAI, Josh Achiam, Steven Adler, et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  25. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186. Association for…
  26. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 1470–1480. Associ…
  27. Qwen, :, An Yang, Baosong Yang, et al. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  28. Tianze Shi, Chen Zhao, Jordan Boyd-Graber, et al. 2020. On the potential of lexico-logical alignments for semantic parsing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1849–1864. Association for Computational Linguistics. find
  29. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM), pages 645–654. Associatio… find
  30. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7567–7578. Association for Co…
  31. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In The Twelfth International Conference… find
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), pages 1–14.
  33. Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. 2024. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174. — [arXiv:2408.09174](https://arxiv.org/abs/2408.09174) find
  34. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8413–8426. Association for Computational Linguistics…
  35. Tao Yu, Rui Zhang, Kai Yang, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3911–3921. Association for Computational…
  36. Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. 2024a. TableLlama: Towards open large generalist models for tables. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 6024–6044. Associatio… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-tableeval-real-world-benchmark-complex/