Elamur Research⌕
☰
home / papers / 2021 / HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation

HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation

Paper · 2021 · Zhoujun Cheng, Haoyu Dong, Zhiruo Wang et al.

Summary

Most table reasoning research assumes flat tables. Real statistical and scientific tables, however, often have multi-level headers, unmarked totals and proportions, and implicit links between entities. HiTab is a cross-domain English benchmark built to test question answering (QA) and table-to-text generation (NLG) on such hierarchical tables. Its tables come from StatCan and NSF HTML statistical reports, plus hierarchical tables drawn from ToTTo (Wikipedia). The texts are not invented by annotators: they are expert-written report sentences that annotators split, cleaned, and turned into QA pairs. Annotators also linked entity and quantity mentions to cells and wrote spreadsheet formulas to record how composite numbers are computed.

Existing baselines struggle on HiTab. The authors propose a hierarchy-aware logical form and partially supervised training that uses the alignment annotations, and together these sharply raise QA accuracy. For NLG, conditioning on both highlighted cells and calculation operators gives better descriptions. Scores still drop as header depth grows and under cross-domain splits, so HiTab remains a hard testbed for complex tables.

Key points

  • Size: 3,597 tables and 10,672 annotated sentences/QA pairs. About half the tables (1,851) come from ToTTo, and the rest from 1,083 StatCan and 208 NSF reports.
  • Annotation effort: 18 trained annotators worked about 2,400 hours. Inter-annotator agreement (Fleiss Kappa) was 0.89 for quantity alignment and 0.82 for entity alignment.
  • QA results: MAPO with a flat-table logical form reaches only 29.2% accuracy. The hierarchy-aware logical form with partial supervision lifts this to 45.1%.
  • NLG results: T5 performs best, scoring 19.5 BLEU-4 and 35.7 PARENT with cell highlights only. Adding calculation types raises this to 26.6 BLEU-4 and 36.9 PARENT, and quality declines as hierarchy depth increases.
  • Availability: Data and code are released at github.com/microsoft/HiTab.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏢Microsoft Research AsiaBeijing · China82 works · 61,484 citations🎓Xi'an Jiaotong UniversityXi'an · China42 works · 36,431 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 487 worksText generation · 214 works

Cite

Cite as Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou and Dongmei Zhang. 2021. HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation. arXiv:2108.06712.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.06712 · arXiv 2108.06712

@misc{cheng2021hitab,
  title = {{HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation}},
  author = {Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou and Dongmei Zhang},
  year = {2021},
  doi = {10.48550/arxiv.2108.06712},
  eprint = {2108.06712},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.06712}
}

References

56 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eva Banik, Claire Gardent, and Eric Kow. The kbgen challenge. In the 14th European Workshop on Natural Language Generation (ENLG), pages 94– 97, 2013. find
  2. BLS. U.s. bureau of labor statistics. https://www.bls.gov Accessed July 4, 2021. ↗
  3. CDC. Centers for disease control and prevention. https://www.cdc.gov Accessed July 4, 2021. ↗
  4. Census. Census bureau. https://www.census.gov . Accessed July 4, 2021. ↗
  5. Zhe Chen and Michael Cafarella. Integrating spreadsheet data via accurate and low-effort extraction. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1126–1135, 2014. find
  6. David L Chen and Raymond J Mooney. Learning to sportscast: a test of grounded language acquisition. In Proceedings of the 25th international conference on Machine learning, pages 128–135, 2008. find
  7. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. Logical natural language generation from open-domain tables. arXiv preprint arXiv:2004.10404, 2020.
  8. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347, 2020.
  9. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122, 2021.
  10. Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, 1977. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  12. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. Handling divergent reference texts when evaluating table-to-text generation. arXiv preprint arXiv:1906.01081, 2019.
  13. Sebastian Gehrmann, Falcon Z Dai, Henry Elder, and Alexander M Rush. End-to-end content and plan selection for data-to-text generation. arXiv preprint arXiv:1810.04700, 2018. find
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324, 2018.
  15. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, 2020.
  16. Yusra Ibrahim, Mirek Riedewald, Gerhard Weikum, and Demetrios Zeinalipour-Yazti. Bridging quantities in tables and text. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pages 1010–1021. find
  17. IMF. International monetary fund. https://www.imf.org . Accessed July 4, 2021. ↗
  18. J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics, pages 159–174, 1977. find
  19. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. arXiv:1603.07771, 2016.
  20. Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking in text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943–6954, 2020. find
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  22. Percy Liang, Michael I Jordan, and Dan Klein. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 91–99, 2009.
  23. Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 23–33, 2017. find
  24. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018. find
  25. Seung-Jin Lim and Yiu-Kai Ng. An automated approach for retrieving hierarchical data from html tables. In Proceedings of the eighth international conference on Information and knowledge management, pages 466–474, 1999. find
  26. Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. Learning to reason for text generation from scientific tables. arXiv:2104.08296, 2021. find
  27. Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871, 2020.
  28. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. Crowd-sourcing nlg data: Pictures elicit better data. arXiv preprint arXiv:1608.00339, 2016. find
  29. NSF. National science foundation. https://www.nsf.gov . Accessed July 4, 2021. ↗
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  31. Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373, 2020.
  32. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015.
  33. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026–8037, 2019.
  34. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042, 2018.
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683, 2019.
  36. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280, 2020.
  37. Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1162–1167. find
  38. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017. find
  39. Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to sql queries. arXiv:2010.11246, 2020. find
  40. StatCan. Statistics canada. https://www150.statcan.gc.ca . Accessed July 4, 2021. ↗
  41. Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… find
  42. Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. Table cell search for question answering. In Proceedings of the 25th International Conference on World Wide Web, pages 771–782, 2016. find
  43. Avijit Thawani, Jay Pujara, Pedro A Szekely, and Filip Ilievski. Representing numbers in nlp: a survey and a vision. arXiv preprint arXiv:2103.13136, 2021.
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.
  45. Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332–1342, 2015. find
  46. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. Structure-aware pre-training for table understanding with tree-based transformers. arXiv:2010.12537, 2020. — [arXiv:2010.12537](https://arxiv.org/abs/2010.12537) find
  47. Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. Retrieving complex tables with multi-granular graph representation learning. arXiv preprint arXiv:2105.01736, 2021. — [arXiv:2105.01736](https://arxiv.org/abs/2105.01736) find
  48. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  49. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. Challenges in data-to-document generation. arXiv preprint arXiv:1707.08052, 2017.
  50. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314, 2020.
  51. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887, 2018.
  52. Yuchen Zhang, Panupong Pasupat, and Percy Liang. Macro grammars and holistic triggering for efficient semantic parsing. arXiv preprint arXiv:1707.07806, 2017.
  53. Xingyao Zhang, Linjun Shou, Jian Pei, Ming Gong, Lijie Wen, and Daxin Jiang. A graph representation of semi-structured data for web question answering. arXiv preprint arXiv:2010.06801, 2020. find
  54. Xuanyu Zhang. Cfgnn: Cross flow graph neural networks for question answering on complex tables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9596–9603, 2020. find
  55. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv:1709.00103, 2017.
  56. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2021-hitab-hierarchical-table-dataset-question/