Elamur Research
☰
home / papers / 2019 / BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1300.pdf

Summary

Most entailment benchmarks depend on crowd workers writing hypotheses, which tends to produce examples that shallow cues can solve. This paper tests inference with yes/no questions instead: questions real users typed into Google search without being prompted and without knowing the answer. The authors reuse the Natural Questions (NQ) annotation pipeline and add a filter for yes/no queries. The result is BoolQ, about 16k questions. Each is paired with a Wikipedia paragraph that an annotator judged sufficient to answer it, and each is labeled yes or no. Their analysis shows that many questions ask for non-factoid information and need entailment-style reasoning across a whole paragraph. They then compare transfer-learning baselines built on entailment, paraphrase, extractive QA and unsupervised pre-training. Crowd-sourced NLI data, especially MultiNLI, transfers best, and it still helps on top of BERT. Fine-tuning BERT-large on MultiNLI and then on BoolQ reaches 80.4% test accuracy, compared with 62% for the majority class and 90% for humans. BoolQ is therefore a hard benchmark tied to a real task, and supervised entailment data appears to add something that language-model pre-training lacks.

Key points

  • Data source: questions come from anonymized Google search queries that begin with one of 13 indicator words ("is", "does", "can" and so on) and that return a Wikipedia page among the top five results. Each question is paired with an annotator-selected passage rather than the full article, and the article title is also provided.
  • Test accuracy: majority class 62.31%; recurrent model + MultiNLI 67.52%; BERT-large + MultiNLI 80.43%. Human annotators reach about 90%.
  • MultiNLI on top of BERT: pre-training BERT-large on MultiNLI before BoolQ fine-tuning adds about 3.5 points. Removing the contradiction class from MultiNLI drops dev accuracy to 78.43%, close to plain BERT-large. The authors conclude that contradiction examples are the key ingredient.
  • Datasets in a similar format help little: QA-style data with multi-sentence passages (SQuAD 2.0, RACE, and a yes/no subset of MS Marco) transferred poorly. NQ, QNLI, QQP and SNLI gave smaller gains than MultiNLI.
  • Small training sets: with only 1,000 training examples, MultiNLI pre-training improves BERT-large by nearly 10 points. At that size, the recurrent model with MultiNLI even beats BERT-large without it. Data and code were released at goo.gl/boolq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

985 papers of our library cite it (of the 140673 whose bibliographies we read); 66 of them have a page here, 919 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1300 · arXiv 1905.10044

@inproceedings{clark2019boolq,
  title = {{BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions}},
  author = {Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1300},
  eprint = {1905.10044},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1300}
}

References

41 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of the IEEE international conference on computer vision. find
  2. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A Textual Entailment Dataset from Science Question Answering. In Proceedings of AAAI. find
  3. Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. ICLR. find
  4. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2011. The Seventh PASCAL Recognizing Textual Entailment Challenge. In TAC. find
  5. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. EMNLP. find
  7. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-Scale Reading Comprehension Dataset from Examinations. EMNLP. find our card
  8. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The Sixth PASCAL Recognizing Textual Entailment Challenge. In TAC. find
  9. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In ACL. find
  10. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuaC: Question Answering in Context. EMNLP. find our card
  11. Alexis Conneau and Douwe Kiela. 2018. Senteval: An Evaluation Toolkit for Universal Sentence Representations. In LREC. find our card
  12. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv:1809.02922. Version 2. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. Version 1. find
  14. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. ACL. find our card
  15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. NAACL. find
  16. Minghao Hu, Yuxing Peng, Zhen Huang, Nan Yang, Ming Zhou, et al. 2018. Read+ Verify: Machine Reading Comprehension with Unanswerable Questions. CoRR. find
  17. Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. EMNLP. find our card
  18. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL. find our card
  19. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Computing Research Repository, arXiv:1902.01007. Version 1. find our card
  20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP. find
  21. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  22. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. arXiv:1611.09268. Version 3. find our card
  23. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In EMNLP. find our card
  24. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL. find
  25. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. Computing Research Repository, arXiv:1811.01088. Version 2. find our card
  26. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In EMNLP. find our card
  27. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. find
  28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. ACL. find our card
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ Questions for Machine Comprehension of Text. EMNLP. find our card
  30. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A Conversational Question Answering Challenge. In TACL. find our card
  31. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of Natural Language Rules in Conversational Machine Reading. In EMNLP. find
  32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional Attention Flow for Machine Comprehension. In ICLR. find our card
  33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. find our card
  34. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In ACL. find our card
  35. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In ICLR. find our card
  36. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NACL. find
  37. Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2017. Visual Question Answering: A Survey of Methods and Datasets. In Computer Vision and Image Understanding. Elsevier. find
  38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  39. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In EMNLP. find
  40. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension. Computing Research Repository, arXiv:1810.12885. Version 1. find our card
  41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card