INFOTABS: Inference on Tables as Semi-structured Data
Paper · 2020 · Vivek Gupta, Maitrey Mehta, Pegah Nokhiz et al.
Summary
Gupta et al. ask whether modern NLI systems can reason over semi-structured text rather than ordinary sentences. They introduce INFOTABS, an English inference benchmark whose premises are Wikipedia info-boxes: heterogeneous key-value tables with no fixed schema. Hypotheses are short sentences written by Amazon Mechanical Turk workers, and each pair is labelled as entailment, contradiction or neutral. Deciding a label often means combining several rows with lexical knowledge, temporal or numerical understanding, and common sense. To separate real reasoning from memorised artifacts, the authors provide three test sets:
- α1, drawn from the same distribution as training;
- α2, an adversarial set where experts minimally edit hypotheses to flip their labels;
- α3, a cross-domain set built from table categories absent in training.
Human annotators largely agree on the labels. However, word-overlap baselines, structure-aware models and BERT/RoBERTa-based NLI systems all fall short across the three test sets. The resource therefore offers a demanding testbed for models that treat tables and similar semi-structured data as first-class inputs.
Key points
- INFOTABS has 23,738 premise–hypothesis pairs built on 2,540 unique Wikipedia info-boxes from many categories. Tables with fewer than 3 rows, non-English cells, specialist content or images were excluded.
- The split sizes are:
- Train: 1,740 tables and 16,538 pairs.
- Dev, α1, α2 and α3: 200 tables and 1,800 pairs each.
- Annotation followed a fixed design:
- Three turkers wrote one true, one false and one undetermined sentence per table, giving nine hypotheses per table.
- Workers were paid 50¢ per HIT, with a 10% bonus for multi-row hypotheses.
- In α2, entailments and contradictions are swapped through minimal word edits while neutrals stay unchanged. α3 tests transfer to unseen domains.
- Compared with TabFact, INFOTABS uses heterogeneous rather than column-homogeneous tables, adds a neutral class and covers broader reasoning types. Data and scripts are available at infotabs.github.io.
Institutions
United States📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works
Cite
Cite as Vivek Gupta, Maitrey Mehta, Pegah Nokhiz and Vivek Srikumar. 2020. INFOTABS: Inference on Tables as Semi-structured Data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
@inproceedings{gupta2020infotabs,
title = {{INFOTABS: Inference on Tables as Semi-structured Data}},
author = {Vivek Gupta, Maitrey Mehta, Pegah Nokhiz and Vivek Srikumar},
booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
year = {2020},
doi = {10.18653/v1/2020.acl-main.210},
url = {https://doi.org/10.18653/v1/2020.acl-main.210}
}References
- Ron Artstein and Massimo Poesio. 2008. Inter-coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555–596. find
- Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. In International Conference on Learning Representations. our card
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
- Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. TabFact : A Large-scale Dataset for Table-based Fact Verification. In International Conference on Learning Representations. our card
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. our card
- Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling Divergent Reference Texts when Evaluating Table-to-Text Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. our card
- Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research. find
- Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. our card
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… our card
- Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question Answering via Integer Programming over Semi-structured Knowledge. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. find
- Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A Textual Entailment Dataset from Science Question Answering. In Association for the Advancement of Artificial Intelligence. find
- Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From Word Embeddings to Document Distances. In International Conference on Machine Learning. find
- Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. find
- Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring Numeracy in Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
- Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics. find
- Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A New Benchmark for Natural Language Understanding. arXiv preprint arXiv:1910.14599. find
- Timothy Niven and Hung-Yu Kao. 2019. Probing Neural Network Comprehension of Natural Language Arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
- Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. our card
- Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. our card
- Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. find
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI Blog. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. our card
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
- Chapter of the Association for Computational Linguistics: Human Language Technologies. find
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. ArXiv. our card
- Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
- Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o… find
- A Figure 4 shows two additional examples of table premises and their corresponding hypotheses available in the development set of I NFO T AB S. Kamloops Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a. Universal Adversarial Triggers for Attacking and Analyzing NLP. In P… find
- Type Mayor Governing body MP MLAs Elected city council Ken Christian Kamloops City Council Cathy McLeod Peter Milobar, Todd Stone H1: Kamloops has a democracy structure. H2: If Ken Christian resigns as Mayor of Kamloops then Cathy McLeod will most likely replace him. H3: Kamloops is ruled by a presi… find
- Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019b. Do NLP Models Know Numbers? Probing Numeracy in Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… our card
- Jefferson Starship Origin Genres Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems. our card
- Years active Labels Associated acts Website San Francisco California Rock, hard rock, psychedelic rock, progressive rock, soft rock 1970 - 1984, 1992 - present RCA Grunt Epic Jefferson Airplane Starship, KBC Band, Hot Tuna www.jeffersonstarship.net H1: Jefferson Starship was started on the West Coas… find
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b. GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations. our card
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Examples of Data *Figure 4: Two semi-structured premises (the tables), and three hypotheses (H1: entailm… find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2020-infotabs-inference-tables-as-semi/