Elamur Research
☰
home / papers / 2021 / TAPEX: Table Pre-training via Learning a Neural SQL Executor

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2107.07653

Summary

Pre-training language models on tables is hard because large, clean table corpora are scarce. Web-crawled table–text pairs are noisy, and template-based synthesis is costly and repetitive. TAPEX avoids this problem by training a model to act like a SQL engine. The authors automatically sample executable SQL queries over existing tables, run them to get the correct outputs, and then continue pre-training BART (Large configuration) to generate those outputs from the query and a flattened table. Because SQL can be sampled systematically, the synthetic corpus is cheap, varied and free of label errors. For downstream use, table question answering is cast as generation and fact verification as classification on the decoder, both over the same input format. On four benchmarks (weakly supervised WikiSQL, WikiTableQuestions, SQA and TabFact), TAPEX sets new state-of-the-art results and clearly beats earlier table pre-training methods. The work shows that learning to execute formal programs is an efficient way to teach table understanding without mined data. Code is available at github.com/microsoft/Table-Pretraining.

Key points

  • Reported results: WikiSQL-Weak denotation accuracy 89.5% (+2.3), WikiTableQuestions 57.5% (+4.8), SQA 74.5% (+3.5), TabFact 84.2% (+3.2). On WikiTableQuestions, TAPEX gains 19.5 absolute points over plain BART.
  • Input format: the NL sentence comes first, followed by the linearized table. The table is marked with [HEAD] and [ROW] tokens, row indices and | column separators. QA answers are decoded as comma-separated strings, and TabFact uses a binary classifier on the last decoder token. Both single-task and multi-task fine-tuning are explored.
  • Ablation on the corpus language: SQL queries in a 0.5M-example corpus were translated into NL with a BART-Large SQL-to-NL model trained on SQUALL (about 9,000 pairs). Only about 68% of the translations were faithful, and pre-training on them did not help (WikiTableQuestions dev: 52.8 vs. 54.2 with SQL).
  • Effect of query difficulty: adding harder SQL templates improves accuracy on questions of matching difficulty. Pre-training with queries up to Medium level already lifts Hard WikiTableQuestions questions by up to 13.1 points over BART.
  • Execution accuracy: on 20,000 held-out SQL queries, the model's per-operator accuracy ranges from about 84% to 91%. For example, Filter reaches 90.6% and Sort 84.1%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 159 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.07653 · arXiv 2107.07653

@misc{liu2021tapex,
  title = {{TAPEX: Table Pre-training via Learning a Neural SQL Executor}},
  author = {Liu, Qian and Chen, Bei and Guo, Jiaqi and Ziyadi, Morteza and Lin, Zeqi and Chen, Weizhu and Lou, Jian-Guang},
  year = {2021},
  doi = {10.48550/arxiv.2107.07653},
  eprint = {2107.07653},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.07653}
}

References

52 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In ICML, 2019. find
  2. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, M. Zhou, and H. Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. In ICML, 2020. find our card
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  4. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  5. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2020. URL https://openreview. net/forum?id=rkeJRhNYDH. find our card
  6. Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a… find
  7. Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. TURL: table understanding through representation learning. Proc. VLDB Endow., 14(3):307–319, 2020. doi: 10.5555/3430915.3442430. URL http://www.vldb.org/pvldb/vol14/p307-deng.pdf. — [doi:10.5555/3430915.3442430](https://doi.org/10.5555/3430915.3442430) ↗
  8. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
  10. Julian Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 281–296, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.… find
  11. Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. Weakly supervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1809–1819, Melbourne, Australia, July 2… find
  12. Jiaqi Guo, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. Weakly supervised semantic parsing by learning from mistakes. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2603–2617, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL… find
  13. Tonglei Guo and Huilin Gao. Using database rule for weak supervised text-to-sql generation. ArXiv, abs/1907.00620, 2019. — [arXiv:1907.00620](https://arxiv.org/abs/1907.00620) find
  14. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4320–4333, Online, July 2020. Association for C… find our card
  15. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1821–1831, Vancouver, Canada, July 2017. Association for Co… find our card
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Assoc… find our card
  17. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of NIPS, 2018. find
  18. Qian Liu, Bei Chen, Haoyan Liu, Jian-Guang Lou, Lei Fang, Bin Zhou, and Dongmei Zhang. A split-and-recombine approach for follow-up query analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… find
  19. Qian Liu, Bei Chen, Jiaqi Guo, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. How far are we from effective context modeling? an exploratory study on semantic parsing in context twitter. In IJCAI, 2020. find
  20. Qian Liu, Dejian Yang, Jiahui Zhang, Jiaqi Guo, Bin Zhou, and Jian-Guang Lou. Awakening latent grounding from pretrained language models for semantic parsing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1174–1189, Online, August 2021. Association for Computatio… find
  21. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  22. Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, and Yasemin Altun. Answering conversational questions on structured data without logical forms. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat… find
  23. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016… find
  24. Arvind Neelakantan, Quoc V. Le, Martı́n Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. find
  25. OpenReview.net, 2017. URL https://openreview.net/forum?id=ry2YOrcge. ↗
  26. Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. arXiv preprint arXiv:2012.14610, 2020. find
  27. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations… find our card
  28. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470–1480, Bei… find
  29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67, 2020. find
  30. Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cı́cero Nogueira dos Santos, and Bing Xiang. Learning contextual representations for semantic parsing with generation-augmented pre-training. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Th… find
  31. AAAI Press, 2021a. URL https: //ojs.aaai.org/index.php/AAAI/article/view/17627. find
  32. Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. Learn to combine linguistic and symbolic information for table-based fact verification. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5335–5346, Barcelona, Spain (Online), December 2020a. International Committee on C… find
  33. Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. Logic-level evidence retrieval and graph-based verification network for table-based fact verification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 175–184, Online and Punta Cana, Dominican Republic, Novemb… find
  34. Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1849–1864, Online, November 2020b. Association for Computationa… find
  35. Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, X. Feng, and Bing Qin. Knowledge-aware conversational semantic parsing over web tables. In NLPCC, 2019. find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neur… find
  37. Bailin Wang, Ivan Titov, and Mirella Lapata. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… find
  38. Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. Learning to synthesize data for semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2760– 2766, Online, June 2021a. Assoc… find
  39. Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Xin Mao, Oleksandr Polozov, and Rishabh Singh. Robust text-to-sql generation with execution-guided decoding. ArXiv, abs/1807.03100, 2018. find
  40. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810–1822, Florence, Italy, July 2019b. Association fo… find our card
  41. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. TUTA: tree-based transformers for generally structured table pre-training. In Feida Zhu, Beng Chin Ooi, and Chunyan Miao (eds.), KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Eve… find
  42. ACM, 2021b. doi: 10.1145/3447548.3467434. URL https://doi.org/10.1145/3447548.3467434. — [doi:10.1145/3447548.3467434](https://doi.org/10.1145/3447548.3467434) ↗
  43. Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, and Xiaodan Zhu. Program enhanced fact verification with verbalization and graph attention network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7810–7825, Online, November 2020. Asso… find
  44. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8413–8426, Online, July 2020. Association for Computational Lingui… find our card
  45. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Confer… find our card
  46. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. Grappa: Grammar-augmented pre-training for table semantic parsing. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=ky… ↗
  47. Tao Yu, Rui Zhang, Alex Polozov, Christopher Meek, and Ahmed Hassan Awadallah. Score: Pre-training for context representation in conversational semantic parsing. In International Conference on Learning Representations, 2021b. URL https://openreview.net/forum?id=oyZxhRI2RiE. ↗
  48. Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. Table fact verification with structure-aware transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1624–1629, Online, November 2020. Association for C… find
  49. Yuchen Zhang, Panupong Pasupat, and Percy Liang. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1214–1223, Copenhagen, Denmark, September 2017. Association for Computational Linguis… find
  50. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017. find
  51. Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6869–6882, Online, November 2020a. Association for Computational Linguis… find
  52. Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. LogicalFactChecker: Leveraging logical operations for fact checking with graph module network. In Proceedings of the 58th Annual Meeting of the Association for Computational L… find