Elamur Research⌕
☰
home / papers / 2020 / Understanding tables with intermediate pre-training

Understanding tables with intermediate pre-training

Paper · 2020 · Julian Eisenschlos, Syrine Krichene and Thomas Müller

Summary

Table entailment means deciding whether a sentence is supported or contradicted by a table. It requires understanding language and table layout together, plus numeric and discrete operations such as comparisons and aggregations. On TabFact, models still trail humans by a wide margin. The authors convert TAPAS, a BERT-style model with extra row, column and rank embeddings, into a binary entailment classifier. They add an intermediate pre-training stage between MASK-LM pre-training and fine-tuning. For this stage they automatically generate millions of balanced examples from Wikipedia tables of two kinds:

  • Counterfactual pairs: an entity in a real sentence about a table is replaced by a different value from the same column.
  • Synthetic statements: sampled from a small grammar of SQL-like filters, aggregations and comparisons.

They also test column pruning to shorten inputs. The approach sets new state-of-the-art results on TabFact and on the sequential QA dataset SQA, needs relatively little labelled data, and runs considerably faster with pruning. The work shows that cheap, automatically generated reasoning data can substantially strengthen table understanding.

Key points

  • The pre-training corpus comes from a December 2019 Wikipedia dump. Tables were split until each had at most 50 cells, giving about 3.7 million tables. From these, 4.1 million counterfactual pairs were extracted, 546K of them with a "supporting mention."
  • A manual check of 100 counterfactual negatives found that 82% were truly refutable when a supporting mention existed, versus 22% without one. The weaker examples still improved end-task accuracy.
  • On TabFact, the method beats earlier systems by about 6 points with BERT-base and 9 points with BERT-large. Using only 10% of the training data, it matches previous approaches.
  • The same intermediate pre-training raises the SQA state of the art by roughly 4 points. It also makes results more consistent across runs: examples solved by all 9 models rose from 24.2% (MASK-LM) to 55.5%.
  • Column pruning doubles inference speed for under 1 accuracy point and gives a 4× speed-up at a larger cost. Code and checkpoints are released at github.com/google-research/tapas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🏢Google (Switzerland)Zurich · Switzerland10 works · 27,513 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Julian Eisenschlos, Syrine Krichene and Thomas Müller. 2020. Understanding tables with intermediate pre-training. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.27

@inproceedings{eisenschlos2020understanding,
  title = {{Understanding tables with intermediate pre-training}},
  author = {Julian Eisenschlos, Syrine Krichene and Thomas Müller},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.27},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.27}
}

References

52 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the Association for Computational Linguistics, pages 6168–6173, Florence, Italy. Association for Computational Linguistics. our card
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. find
  3. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia. find
  4. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia. our card
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 2924–2936, Minnea… our card
  6. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia. find
  7. Peter Clark. 2019. Project aristo: Towards machines that capture and reason with science knowledge. In Proceedings of the 10th International Conference on Knowledge Capture, K-CAP âĂŹ19, page 1âĂŞ2, New York, NY, USA. Association for Computing Machinery. find
  8. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association fo… our card
  9. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rationale, evaluation and approaches. Journal of Natural Language Engineering, 4. find
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186, Minneapolis, Minnesota. Association… our card
  12. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of Empirical Methods in Natural Language Processing, pages 2185–2194, Hong Kong, China. Association for Computational Linguistics. find
  13. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noa… our card
  14. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2019. Posing fair generalization tasks for natural language inference. In Proceedings of Empirical Methods in Natural Language Processing, pages 4485–4495, Hong Kong, China. Association for Computational Linguistics. our card
  15. Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In acl, pages 946–958, Online. Association for Computational Linguistics. our card
  16. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. Google Vizier: A Service for Black-Box Optimization. find
  17. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. Infotabs: Inference on tables as semi-structured data. In Proceedings of the Association for Computational Linguistics, Seattle, Washington. Association for Computational Linguistics. our card
  18. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the Association for Computational Linguistics, Seattle, Washington. Association for Computational Li… find
  19. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Association for the Advancement of Artificial Intelligence. find
  20. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Computational Linguistics. our card
  21. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the Association for Computational Linguistics, pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  22. Shankar Iyer, Nikhil Dandekar, , and Kornél Csernai. 2017. Quora question pairs. find
  23. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the Association for Computational Linguistics, pages 1821– 1831, Vancouver, Canada. Association for Computational Linguistics. our card
  24. Saehan Jo, Immanuel Trummer, Weicheng Yu, Xuezhi Wang, Cong Yu, Daniel Liu, and Niyati Mehta. 2019. Aggchecker: A fact-checking system for text summaries of relational data sets. International Conference on Very Large Databases, 12(12):1938âĂŞ1941. find
  25. Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020. Learning the difference that makes A difference with counterfactually-augmented data. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia. find
  26. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. In International Joint Conference on Artificial Intelligence, IJCAIâĂŹ16, page 1145âĂŞ1152. AAAI Press. find
  27. Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485–525. our card
  28. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. our card
  29. Rezka Leonandya, Dieuwke Hupkes, Elia Bruni, and Germán Kruszewski. 2019. The fast and the flexible: Training neural networks to learn to follow instructions from small data. In Proceedings of the International Conference on Computational Semantics, pages 223–234, Gothenburg, Sweden. Association for… find
  30. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the Association for Computational Linguistics, pages 4896–4910, Florence, Italy. Association for Computational Linguistics. our card
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  32. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. In Tools and methodologies for teaching. find
  33. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of Advances in Neural Information Processing Systems, NIPSâĂŹ13, page 3111âĂŞ3119, Lake Tahoe, Nevada. Curran Associates Inc. our card
  34. Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, and Yasemin Altun. 2019. Answering conversational questions on structured data without logical forms. In Proceedings of Empirical Methods in Natural Language Processing, pages 5902–5910, Hong Kong, China. Association for Computational… find
  35. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the Association for Computational Linguistics, pages 1470–1480, Beijing, China. Association for Computational Linguistics. our card
  36. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? In Proceedings of the Asso… find
  37. Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. NumNet: Machine reading comprehension with numerical reasoning. In Proceedings of Empirical Methods in Natural Language Processing, pages 2474–2484, Hong Kong, China. Association for Computational Linguistics. find
  38. Felipe Salvatore, Marcelo Finger, and Roberto Hirata Jr. 2019. A logical-based corpus for cross-lingual evaluation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 22–30, Hong Kong, China. Association for Computational Linguistics. find
  39. Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. 2010. “ask not what textual entailment can do for you...”. In Proceedings of the Association for Computational Linguistics, pages 1199–1208, Uppsala, Sweden. Association for Computational Linguistics. our card
  40. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the Association for Computational Linguistics, Seattle, Washington. Association for Computational Linguistics. find
  41. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the Association for Computational Linguistics, pages 455– 465, Sofia, Bulgaria. Association for Computational Linguistics. find
  42. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In Proceedings of the International Conference on Learning Representations. find
  43. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the Association for Computational Linguistics, pages 217–223, Vancouver, Canada. Association for Computational Linguistics. find
  44. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Association for Computational Linguistics, pages 6418–6428, Florence, Italy. Association for Computational Linguistics. our card
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings of Advances in Neural Info… our card
  46. Andreas Vlachos and Sebastian Riedel. 2015. Identification and verification of simple claims about statistical properties. In Proceedings of Empirical Methods in Natural Language Processing, pages 2596– 2601, Lisbon, Portugal. Association for Computational Linguistics. find
  47. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of Empirical Methods in Natural Language Processing, pages 5307–5315, Hong Kong, China. Association for Computational Linguistics. our card
  48. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the Association for Computational Linguistics, pages 1332–1342, Beijing, China. Association for Computational Linguistics. find
  49. Nathaniel Weir, Prasetya Utama, Alex Galakatos, Andrew Crotty, Amir Ilkhechi, Shekar Ramaswamy, Rohin Bhushan, Nadja Geisler, Benjamin Hättasch, Steffen Eger, Ugur Cetintemel, and Carsten Binnig. 2020. Dbpal: A fully pluggable nl2sql training pipeline. In Proceedings of the ACM SIGMOD International… find
  50. Changxing Wu, Xiaodong Shi, Yidong Chen, Yanzhou Huang, and Jinsong Su. 2016. Bilingually-constrained synthetic data for implicit discourse relation recognition. In Proceedings of Empirical Methods in Natural Language Processing, pages 2306– 2312, Austin, Texas. Association for Computational Linguis… our card
  51. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia. find
  52. Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. 2020. Logical-FactChecker: Leveraging logical operations for fact checking with graph module network. In Proceedings of the Association for Computational Linguistics, Seattle,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eisenschlos-2020-understanding-tables-intermediate-pre-training/