Elamur Research⌕
☰
home / papers / 2025 / TANQ: An Open Domain Dataset of Table Answered Questions

TANQ: An Open Domain Dataset of Table Answered Questions

Paper · 2025 · Mubashara Akhtar, Chenxi Pang, Andreea Marzoca et al.

Summary

Most question-answering benchmarks for LLMs expect a short string answer drawn from a single document or table. Real information-seeking work often means collecting facts scattered across many sources and organizing them into a structured result. TANQ addresses this gap. It is an open-domain, multi-hop dataset in which each answer is a table assembled from Wikipedia text, tables and infoboxes.

The authors started from QAMPARI questions, which have lists of entities as answers. They extended these questions with extra Wikidata relations, gathered supporting Wikipedia evidence and built gold tables. They then rephrased the questions and added harder skills such as filtering, arithmetic and unit conversion. The pipeline is largely automatic, with PaLM-2 checking intermediate steps, and every cell is linked to its source evidence.

The authors evaluated several LLMs in closed-book, oracle and open-book settings. Even the strongest system clearly trails humans. Typical errors include incomplete tables, ignored filter conditions and confusion on composition or intersection questions. TANQ therefore offers a demanding testbed for retrieval, aggregation and structured generation.

Key points

  • Dataset construction: TANQ comes from a five-step pipeline seeded with QAMPARI, using Wikidata and Wikipedia as knowledge sources. Each answer cell carries source attribution from text, a table or an infobox. The data is hosted at github.com/google-deepmind/tanq.
  • Best result: Gemini Flash in the oracle setting scored highest, with an overall F1 of 60.7. This is 12.3 points below human performance.
  • Open-book baseline: The ReAct-based tool-using model did worse than the other baselines, including the closed-book ones, especially on simple questions.
  • Error analysis on 50 open-book outputs:
    • Filters were ignored in 10 cases.
    • Attributes were missing in 6 cases.
    • Intersection conditions were only partly satisfied in 5 cases.
    • Intermediate entities were returned instead of the requested ones in 4 cases.
  • Evaluation and model size: Scoring uses an adapted version of the RMS metric from Liu et al. (2023). Gemma (9B) performed surprisingly close to larger models such as PaLM-2, Gemini Pro and GPT-4o.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland72 works · 5,176 citations🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

Cite

Cite as Mubashara Akhtar, Chenxi Pang, Andreea Marzoca, Yasemin Altun and Julian Martin Eisenschlos. 2025. TANQ: An Open Domain Dataset of Table Answered Questions. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00749

@article{akhtar2025tanq,
  title = {{TANQ: An Open Domain Dataset of Table Answered Questions}},
  author = {Mubashara Akhtar, Chenxi Pang, Andreea Marzoca, Yasemin Altun and Julian Martin Eisenschlos},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl\_a\_00749},
  url = {https://doi.org/10.1162/tacl_a_00749}
}

References

28 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simperl. 2023. Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15…
  2. Samuel Joseph Amouyal, Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. QAMPARI: An open-domain question answering benchmark for questions with many answers from multiple paragraphs. CoRR, abs/2205.12665. https://doi.org/10.48550/ARXIV.2205.12665 — [doi:10.48550/arxiv.2205.12665](https://doi.org/10.48550/arxiv.2205.12665) ↗
  3. H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddharth… — PaLM 2 Technical Report
  4. Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluı́s Gómez, Marçal Rusiñol, Minesh Mathew, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019. find
  5. ICDAR 2019 competition on scene text visual question answering. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20–25, 2019, pages 1563–1570. IEEE. https://doi.org/10.1109/ICDAR.2019.00251 — [doi:10.1109/icdar.2019.00251](https://doi.org/10.1109/icdar.2019.00251) ↗
  6. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,… — Gemini: A Family of Highly Capable Multimodal Models
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  8. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021a. Open question answering over tables and text. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3–7, 2021. OpenReview.net. find
  9. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020a. Tabfact: A…
  10. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics. https://doi .org/…
  11. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, Online. Association for Computation…
  12. Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. 2023. RobustQA: Benchmarking the robustness of domain adaptation for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 42… find
  13. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021b. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natura…
  14. Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Eisenschlos. 2021. Open domain question answering over tables via dense retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 512…
  15. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279–62…
  16. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada. Association for Computational Lin…
  17. Michael Chui, James Manyika, Jacques Bughin, Richard Dobbs, and Charles Roxburgh. 2012. The social economy: Unlocking value and productivity through social technologies. find
  18. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Fredo Erxleben, Michael Günther, Markus…
  19. Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Ba… find
  20. Amrith Krishna, Sebastian Riedel, and Andreas Vlachos. 2022. ProoFVer: Natural logic theorem proving for fact verification. Transactions of the Association for Computational Linguistics, 10:1013–1030. https://doi.org/10 .1162/tacl_a_00503 find
  21. Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023. DePlot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2…
  22. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022. FeTaQA: Free…
  23. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/…
  24. Vaishali Pal, Andrew Yates, Evangelos Kanoulas, and Maarten de Rijke. 2023. MultiTabQA: Generating tabular answers for multi-table question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6322–6334, Toronto, Canada.… find
  25. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  26. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin,… — ReAct: Synergizing Reasoning and Acting in Language Models
  27. Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277–3287, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.254 ↗
  28. Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022. MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588–6600, Dublin, Ireland. Associat…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akhtar-2025-tanq-open-domain-dataset-table/