Elamur Research⌕
☰
home / papers / 2026 / ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports

ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports

Paper · 2026 · Ruilong Wang and Simone Balloccu

Summary

Enterprise annual reports pair audited KPI tables with narrative explaining why figures changed. Existing QA benchmarks usually test only one ability, such as table arithmetic, hybrid retrieval, or text-only citation. ARQA (Annual Report QA) combines numeric recomputation, joint table–text grounding, and causal explanation in a single benchmark. It contains about 2.5K QA pairs built from ten years of Volkswagen Group annual reports. Each item rests on a "pack", meaning one table plus its related paragraphs, and is annotated with cell- and span-level evidence. Items fall into three families: LOOKUP, ARITHMETIC, and INSIGHT. GPT-4o planned, generated, and self-verified the items. Deterministic checks then filtered them, and domain experts audited the results. The authors tested five instruction-tuned LLMs. The models retrieved facts well but did worse on recomputation, on cross-modal causal explanation, and especially on citing exact text spans. Better prompting helped with procedure but left these gaps open. ARQA therefore works as a diagnostic testbed for auditable, evidence-first reasoning over corporate documents.

Key points

  • Data source: Volkswagen Group annual reports for fiscal years 2015–2024, covering the Production and Management domains. GPT-4o pre-filtered candidate paragraphs (relatedness score ≥ 0.5, pages p±1).
  • Pipeline (plan → generate → verify → audit): GPT-4o was guided by expert-curated focus points. Deterministic checks covered schema validity, cell bounds and values, and exact character spans. ARITHMETIC programs were re-executed at 28-digit precision, with a tolerance of ±1 ULP or a relative error of 10⁻⁶.
  • Expert audit: 35 internal automotive experts each reviewed up to 20 packs. They rated ARITHMETIC and INSIGHT items for Grounded Correctness (pass/fail) and Insight Quality (1–3). Agreement was measured with percent agreement and Gwet's AC1.
  • Models and setups: Llama-3.1 8B Instruct, Llama-3.3 70B Instruct, Qwen-2.5 32B Instruct, DeepSeek-R1-Distill-Qwen 32B, and GPT-4o were each run under two setups:
    • S1: single-pass inference with a schema-explicit prompt.
    • S2: type-aware prompts routed by question family.
    • Both setups were also tested with Few-shot + CoT prompting.
  • Findings:
    • GPT-4o performed best overall, and Qwen-2.5 32B was the strongest open-weight model.
    • S2 raised cell-evidence F1 but lowered value–unit exact match and INSIGHT claim F1.
    • Span evidence F1 stayed very low for every model.
    • The benchmark and toolkit are released on GitHub (RuilongWang/ARQA-Benchmark).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksQuestion answering · 441 works

Cite

Cite as Ruilong Wang and Simone Balloccu. 2026. ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.63

@inproceedings{wang2026arqa,
  title = {{ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports}},
  author = {Ruilong Wang and Simone Balloccu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.63},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.63}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  2. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, Online. Association for Computationa…
  3. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, and 1 others. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Langua…
  4. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279– 6…
  5. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
  6. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  7. Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A benchmark for fact-checking in dialogue. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3785–3801, Dublin, Ireland. Association for Computational…
  8. Kilem Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. The British journal of mathematical and statistical psychology, 61:29–48. find
  9. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. Preprint, arXiv:2311.11944. — [arXiv:2311.11944](https://arxiv.org/abs/2311.11944) find
  10. Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2022. AIT-QA: Question answering dataset over complex tables in the airline industry. In Proceedings of the… find
  11. Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology, 3 edition. SAGE Publications. find
  12. Mark Lang and Lorien Stice-Lawrence. 2014. Textual analysis and international financial reporting: Large sample evidence. SSRN Electronic Journal, 60. find
  13. Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. 2024. AttributionBench: How hard is automatic attribution evaluation? In Findings of the Association for Computational Linguistics: ACL 2024, pages 14919– 14935, Bangkok, Thailand. Association for Computational Linguistics. find
  14. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. Docvqa: A dataset for vqa on document images. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2199–2208.
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  16. Noah Shinn, Saad Labash, and Rohan Gopinath. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366.
  17. Fan Zhou Qingyang Dai Zhixuan Chu Siqiao Xue, Xiaojing Li and Hongyuan Mei. 2024. Famma: A benchmark for financial domain multilingual multimodal question answering. arXiv preprint arXiv:2410.04526. find
  18. Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Christian Biemann, and Martin Semmann. 2025. T2-ragbench: Text-and-table benchmark for evaluating retrieval-augmented generation. ArXiv, abs/2506.12071. — [arXiv:2506.12071](https://arxiv.org/abs/2506.12071) find
  19. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  20. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  22. Siqiao Xue, Xiaojing Li, Fan Zhou, Qingyang Dai, Zhixuan Chu, and Hongyuan Mei. 2024. Famma: A benchmark for financial domain multilingual multimodal question answering. arXiv preprint arXiv:2410.04526. find
  23. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR). find
  24. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-arqa-benchmark-grounded-table-text/