Elamur Research
☰
home / papers / 2020 / TaPas: Weakly Supervised Table Parsing via Pre-training

TaPas: Weakly Supervised Table Parsing via Pre-training

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.398.pdf

Summary

Answering questions over tables is usually treated as semantic parsing: the question is converted into an executable logical form. Training such parsers from answers alone (weak supervision) is hard because many programs can produce the right answer by accident, and the reward signal is sparse. TaPas avoids logical forms altogether. It extends the BERT encoder to read a question followed by a flattened table. Extra embeddings encode the column, row, numeric rank, segment and previous-answer status of each token. One output head chooses table cells, and a second head picks an aggregation operator (NONE, COUNT, SUM, AVERAGE). When the answer needs aggregation, the model is trained by regressing a differentiable expected value against the gold scalar. Before fine-tuning, the model is pre-trained with a masked language model objective on millions of Wikipedia tables paired with nearby text. TaPas sets a new state of the art on the conversational SQA benchmark and matches the best systems on WikiSQL and WikiTQ, while using a simpler encoder-only design with no decoder. This shows that table reasoning can be learned end-to-end from answers alone.

Key points

  • On SQA, accuracy rises from 55.1 to 67.2. Results on WikiSQL and WikiTQ are comparable to the previous best systems.
  • Fine-tuning on WikiSQL and then transferring to WikiTQ gives 48.7 accuracy, 4.2 points above the prior state of the art.
  • The pre-training corpus has 6.2M Wikipedia tables (3.3M Infobox, 2.9M WikiTable, each with at most 500 cells) and 21.3M related text snippets. The final pre-training runs use a 128-token budget.
  • At inference, the model keeps cells whose probability is above 0.5, and it is biased toward selecting cells from a single column. Its limits are that it handles one table that must fit in memory, and only one aggregation per answer.
  • Code and pre-trained models are released at github.com/google-research/tapas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

309 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 301 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.398 · arXiv 2004.02349

@inproceedings{herzig2020tapas,
  title = {{TaPas: Weakly Supervised Table Parsing via Pre-training}},
  author = {Herzig, Jonathan and Nowak, Pawel Krzysztof and Müller, Thomas and Piccinno, Francesco and Eisenschlos, Julian},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.398},
  eprint = {2004.02349},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.398}
}

References

47 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. arXiv preprint arXiv:1902.07198. — [arXiv:1902.07198](https://arxiv.org/abs/1902.07198) find
  2. D. Andor, C. Alberti, D. Weiss, A. Severyn, A. Presta, K. Ganchev, S. Petrov, and M. Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042. find our card
  3. Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving bert a calculator: Finding operations and arguments with reading comprehension. arXiv preprint arXiv:1909.00109. find
  4. Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1:49–62. find our card
  5. J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP). find
  6. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. ArXiv, abs/1909.02164. find our card
  7. Minseok Cho, Reinald Kim Amplayo, Seung won Hwang, and Jonghyuck Park. 2018. Adversarial tableqa: Attention supervision for question answering on tables. In ACML. find
  8. J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world’s response. In Computational Natural Language Learning (CoNLL), pages 18–27. find
  9. A. M. Dai and Q. V. Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NeurIPS). find
  10. Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  12. L. Dong and M. Lapata. 2016. Language to logical form with neural attention. In Association for Computational Linguistics (ACL). find
  13. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL). find our card
  14. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. Google Vizier: A Service for Black-Box Optimization. find
  15. K. Guu, P. Pasupat, E. Z. Liu, and P. Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Association for Computational Linguistics (ACL). find
  16. T. Haug, O. Ganea, and P. Grnarova. 2018. Neural multi-step reasoning for question answering on semi-structured tables. In European Conference on Information Retrieval. find
  17. M. Iyyer, W. Yih, and M. Chang. 2017. Search-based neural structured learning for sequential question answering. In Association for Computational Linguistics (ACL). find our card
  18. R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL). find
  19. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling data. In International Conference on Machine Learning (ICML), pages 282–289. find
  20. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  21. Carolin Lawrence and Stefan Riezler. 2018. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1820–1830, Melbourne, Australia. Association… find
  22. C. Liang, M. Norouzi, J. Berant, Q. Le, and N. Lao. 2018. Memory augmented policy optimization for program synthesis with generalization. In Advances in Neural Information Processing Systems (NeurIPS). find
  23. P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pages 590–599. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  25. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265. find our card
  26. P. J. Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101. find
  27. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  28. Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069. find
  29. D. Muhlgay, J. Herzig, and J. Berant. 2019. Value-based search in execution space for mapping instructions to programs. In North American Association for Computational Linguistics (NAACL). find
  30. S. Iyer, I. Konstas, A. Cheung, J. Krishnamurthy, and L. Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Association for Computational Linguistics (ACL). find
  31. Thomas Müller, Francesco Piccinno, Massimo Nicosia, Peter Shaw, and Yasemin Altun. 2019. Answering conversational questions on structured data without logical forms. arXiv preprint arXiv:1908.11787. find
  32. J. Herzig and J. Berant. 2017. Neural semantic parsing over multiple knowledge-bases. In Association for Computational Linguistics (ACL). find
  33. A. Neelakantan, Q. V. Le, M. Abadi, A. McCallum, and D. Amodei. 2017. Learning a natural language interface with neural programmer. In International Conference on Learning Representations (ICLR). find
  34. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2015. Neural programmer: Inducing latent programs with gradient descent. CoRR, abs/1511.04834. find
  35. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics. find
  36. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1… find
  37. Y. Wang, J. Berant, and P. Liang. 2015. Building a semantic parser overnight. In Association for Computational Linguistics (ACL). find
  38. Pengcheng Yin, Zhengdong Lu, Hang Li, and Kao Ben. 2016. Neural enquirer: Learning to query tables in natural language. In Proceedings of the Workshop on Human-Computer Question Answering, pages 29–35, San Diego, California. Association for Computational Linguistics. find
  39. Y. Zhang, P. Pasupat, and P. Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In Empirical Methods in Natural Language Processing (EMNLP). find
  40. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. find
  41. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). find
  42. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  43. Y. Su and X. Yan. 2017. Cross-domain semantic parsing via paraphrasing. In Empirical Methods in Natural Language Processing (EMNLP). find
  44. Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, Xiaocheng Feng, and Bing Qin. 2018. Knowledge-aware conversational semantic parsing over web tables. arXiv preprint arXiv:1809.04271. — [arXiv:1809.04271](https://arxiv.org/abs/1809.04271) find
  45. Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. find our card
  46. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. arXiv preprint arXiv:1909.07940. find our card
  47. Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find