Elamur Research
☰
home / papers / 2020 / TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.745.pdf

Summary

Pretrained language models such as BERT learn only from running text. They are therefore poorly suited to semantic parsing over databases, where a system must connect a natural-language question to the columns and cell values of a table. TaBERT, built on BERT, addresses this by learning joint representations of utterances and (semi-)structured tables. It was pretrained on a parallel corpus of about 26 million Web tables paired with surrounding English text.

To handle large tables cheaply, TaBERT encodes only a "content snapshot," a few rows chosen by n-gram overlap with the question. Each row is flattened and encoded together with the utterance, and vertical self-attention layers then share information across rows to produce token and column vectors. The model serves as a drop-in encoder for existing neural parsers. With it, parsers set a new best result on the weakly supervised WikiTableQuestions benchmark and stay competitive on the Spider text-to-SQL dataset. This shows that general-purpose pretraining on tables can replace ad-hoc, domain-specific schemes for encoding tables.

Key points

  • Pretraining data: 26 million tables with their English contexts, taken from the Web. Code is released at github.com/facebookresearch/TaBERT.
  • Row linearization: each row is written as a sequence of column name | data type | cell value, separated by [SEP]. Adding type information and snapshot contents improved results over encoding column names alone (the Hwang et al., 2019 style).
  • Content snapshot:
    • With K > 1, the top-K rows by n-gram overlap are selected.
    • With K = 1, a synthetic row is built from the best-matching cell in each column.
    • K = 3 handled questions that need several rows, where K = 1 failed.
  • Pretraining objectives: masked column prediction (MCP) plus cell value recovery (CVR). On dev sets, TaBERT Base (K = 3) scored 62.6 → 63.3 on Spider when CVR was added and 51.6 on WikiTableQuestions either way.
  • Evaluation: tested in both supervised (Spider) and weakly supervised (WikiTableQuestions) settings, where it outperformed BERT-based counterparts. Planned future work includes table retrieval, table-to-text generation, and cross-lingual use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

215 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.745

@inproceedings{yin2020tabert,
  title = {{TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data}},
  author = {Yin, Pengcheng and Neubig, Graham and Yih, Wen-tau and Riedel, Sebastian},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.745},
  url = {https://doi.org/10.18653/v1/2020.acl-main.745}
}

References

52 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In ICML. find
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP. find
  3. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of ACL. find
  4. Ben Bogin, Matt Gardner, and Jonathan Berant. 2019a. Global reasoning over database structures for text-to-sql parsing. ArXiv, abs/1908.11214. find
  5. Ben Bogin, Matthew Gardner, and Jonathan Berant. 2019b. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of ACL. find
  6. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. ArXiv, abs/1909.02164. find our card
  7. Donghyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. ArXiv, abs/2004.03125. find
  8. Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke S. Zettlemoyer, and Eduard H. Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of NAACL-HLT. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. find
  10. Li Dong and Mirella Lapata. 2018. coarse-to-fine decoding for neural semantic parsing. In Proceedings of ACL. find
  11. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. ArXiv, abs/1901.05287. find our card
  12. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of ACL. find
  13. Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. ArXiv, abs/1908.08113. — [arXiv:1908.08113](https://arxiv.org/abs/1908.08113) find
  14. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). ArXiv, abs/1606.08415. find
  15. Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. ArXiv, abs/1902.01069. find
  16. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. In Proceedings of EMNLP. find our card
  17. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of EMNLP. find
  18. Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus of web tables containing time and context metadata. In Proceedings of WWW. find
  19. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of NIPS. find
  20. Jindrich Libovický and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of ACL. find
  21. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019a. K-bert: Enabling language representation with knowledge graph. ArXiv, abs/1909.07606. find our card
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of NIPS. find our card
  24. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of CoNLL. find our card
  25. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In Proceedings of ICLR. find
  26. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of ACL. find
  27. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL. find
  28. Matthew E. Peters, Mark Neumann, IV RobertLLo-gan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of EMNLP. find
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of EMNLP. find our card
  30. Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. Incsql: Training incremental text-to-sql parsers with non-deterministic oracles. ArXiv, abs/1809.05054. — [arXiv:1809.05054](https://arxiv.org/abs/1809.05054) find
  31. Huan Sun, Hao Ma, Xiaodong He, Wen tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of WWW. find
  32. Yibo Sun, Duyu Tang, Nan Duan, Jianshu Ji, Guihong Cao, Xiaocheng Feng, Bing Qin, Ting Liu, and Ming Zhou. 2018. Semantic parsing with syntax-and table-aware SQL generation. In Proceedings of EMNLP. find
  33. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. ArXiv, abs/1904.09223. find our card
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. find
  35. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Margot Richardson. 2019a. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. ArXiv, abs/1911.04942. find
  36. Bailin Wang, Ivan Titov, and Mirella Lapata. 2019b. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In EMNLP/IJCNLP. find
  37. Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Xin Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. ArXiv, abs/1807.03100. find
  38. Daniel C. Wang, Andrew W. Appel, Jeffrey L. Korn, and Christopher S. Serra. 1997. The Zephyr abstract syntax description language. In Proceedings of DSL. find
  39. Zhuosheng Zhang, Yu-Wei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiaodong Zhou. 2019c. Semantics-aware bert for language understanding. ArXiv, abs/1909.02209. — [arXiv:1909.02209](https://arxiv.org/abs/1909.02209) find
  40. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of ACL. find
  41. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv, abs/1709.00103. find
  42. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQL-Net: Generating structured queries from natural language without reinforcement learning. arXiv, abs/1711.04436. find
  43. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of NIPS. find our card
  44. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of ACL. find
  45. Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of EMNLP Demonstration Track. find
  46. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir R. Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-sql generation. In Proceedings of NAACL-HLT. find
  47. Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir R. Radev. 2018b. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of EMNLP. find
  48. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018c. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of EMNLP… find our card
  49. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of AAAI. find
  50. Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2019a. Editing-based sql query generation for cross-domain context-dependent questions. ArXiv, abs/1909.00786. find
  51. Yuchen Zhang, Panupong Pasupat, and Percy Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of EMNLP. find
  52. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. Ernie: Enhanced language representation with informative entities. In Proceedings of ACL. find