Elamur Research⌕
☰
home / papers / 2021 / Open Domain Question Answering over Tables via Dense Retrieval

Open Domain Question Answering over Tables via Dense Retrieval

Paper · 2021 · Jonathan Herzig, Thomas Müller, Syrine Krichene et al.

Summary

Most table QA systems assume the correct table is already provided. Real search engines and assistants, however, must first locate the relevant table in a large collection. This paper is the first to address open-domain QA over tables. It uses a two-stage pipeline: a dense retriever picks candidate tables, and a reader extracts an answer span from them.

The retriever, DTR, uses separate TAPAS encoders for questions and tables. TAPAS adds table-structure embeddings to BERT, and each table is encoded together with its page title. Relevance is scored by the inner product of the two embeddings. DTR is pre-trained on text–table pairs mined from Wikipedia, then fine-tuned with in-batch negatives and extra hard negatives mined by the model itself.

No suitable benchmark existed, so the authors built NQ-TABLES from NATURAL QUESTIONS. Compared with a BERT-initialized retriever, DTR raises recall@10 from 72.0 to 81.1 and end-to-end exact match from 33.8 to 37.7. This shows that encoders aware of table structure matter for retrieval. Code, data generation scripts and models are released in the google-research/tapas repository.

Key points

  • NQ-TABLES dataset: about 11–12K NATURAL QUESTIONS examples whose answers sit in Wikipedia tables. Infobox tables are transposed, and near-duplicate tables are merged by clustering. The split has 9,594 train and 1,068 dev questions, with a retrieval corpus of 169,898 tables.
  • Pre-training data: 21.3M text–table pairs drawn from 6.2M Wikipedia tables. Without this pre-training, R@10 falls to roughly 47.8.
  • Best retriever (DTR +hn): R@1 ≈ 42.4, R@10 ≈ 81.1, R@50 ≈ 92.6. Negatives mined with DTR beat negatives mined with BM25 by about 0.6 R@10. All pre-trained dense models clearly outperform BM25.
  • Reader and significance: a TAPAS-based reader exceeds a BERT reader by more than 3 EM points. DTR +hn is significantly better than all baselines (McNemar, p<0.05).
  • Error analysis on dev: 29% of questions are answered correctly. A further 14% need list answers, which are out of scope. In 12% no retrieved candidate contains the answer, and in 11% the wrong table is chosen. The remaining 34% are reader span errors, often caused by table reasoning or by ambiguity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Tel Aviv UniversityTel Aviv · Israel67 works · 10,006 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 242 worksInformation retrieval & search · 145 works

Cite

Cite as Jonathan Herzig, Thomas Müller, Syrine Krichene and Julian Eisenschlos. 2021. Open Domain Question Answering over Tables via Dense Retrieval. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.43

@inproceedings{herzig2021open,
  title = {{Open Domain Question Answering over Tables via Dense Retrieval}},
  author = {Jonathan Herzig, Thomas Müller, Syrine Krichene and Julian Eisenschlos},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.43},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.43}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada. Association for Computational Li…
  2. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In International Conference on Learning Representations. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281–296, Online. Association for Computational Linguistics.
  5. Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528–537, Hong Kong, China.…
  6. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. Google Vizier: A Service for Black-Box Optimization. find
  7. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119… find
  8. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR. find
  9. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. find
  10. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Compu…
  11. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, pages 1–1.
  12. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl…
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  14. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  15. Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2017. Learning recurrent span representations for extractive question answering. In arXiv 1611.01436. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  16. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. find
  17. Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020. Unified open-domain question answering with structured and unstructured knowledge. arXiv preprint arXiv:2012.14610.
  18. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  19. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta. ELRA. find
  20. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  21. Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231–4242, Brus… find
  22. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
  23. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72…
  24. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018…
  25. Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021. Representations for question answering from documents with tables and text. In Proceedings of the 2021 Conference of the European Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota. Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/herzig-2021-open-domain-question-answering-over/