Know What You Don't Know: Unanswerable Questions for SQuAD

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-2124.pdf

Summary

Extractive question-answering models trained on SQuAD learn to always return some text span. SQuAD guarantees that every question has an answer in the passage, so these models also guess when the passage doesn't support any answer. Earlier datasets that included unanswerable items mostly generated them automatically, for example by retrieving a different paragraph with TF-IDF or by applying rule-based word swaps. Such negatives are usually easy to spot. The authors built SQuADRUN, published as SQuAD 2.0, by having crowdworkers write new unanswerable questions for existing SQuAD paragraphs. Each question had to stay on the paragraph's topic, and the paragraph had to contain a span of the expected answer type that could act as a decoy. A system must therefore answer when the text allows it and abstain when it doesn't. Three neural baselines all lag far behind humans, and the gap is much wider than on the original SQuAD. Models trained and tested with automatically generated negatives score much higher. The resource became a standard benchmark for teaching reading-comprehension systems to recognise when no answer exists.

Key points

Dataset: SQuADRUN adds 53,775 unanswerable questions from the Daemo crowdsourcing platform to the original SQuAD questions, and is released under CC BY-SA 4.0. Training data has about two answerable questions per unanswerable one. The dev and test sets are roughly balanced.
Main results: DocQA + ELMo, the best model tested, reaches 66.3 F1 on the test set, while humans reach 89.5 F1, a gap of 23.2 points. On SQuAD the same architecture scores 85.8 F1, only 5.4 points below humans. A system that always abstains already gets 48.9 F1.
Baselines: BNA, DocQA and DocQA + ELMo each predict a probability that the question has no answer. A threshold tuned on the dev set decides when to abstain.
Automatic negatives are easier: The best score on SQuADRUN dev is 15.4 F1 lower than the best score on SQuAD augmented with TF-IDF or RULEBASED negatives.
Quality and decoys: In a manual check of 100 negatives, 93% were truly unanswerable. They covered negation, antonyms, entity swaps, mutual exclusion and impossible conditions. About half of all wrong answers to unanswerable questions, from both humans and models, matched the crowdworker-provided plausible answer exactly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

1162 papers of our library cite it (of the 140673 whose bibliographies we read); 90 of them have a page here, 1072 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2124 · arXiv 1806.03822

@inproceedings{rajpurkar2018know,
  title = {{Know What You Don't Know: Unanswerable Questions for SQuAD}},
  author = {Rajpurkar, Pranav and Jia, Robin and Liang, Percy},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2124},
  eprint = {1806.03822},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-2124}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. find our card
S. Bowman, G. Angeli, C. Potts, and C. D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP). find
C. Clark and M. Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 . find our card
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). find
S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Ra-jpal, J. Regino, et al. 2015. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. pages 101–102. find
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). pages 193–203. find our card
K. M. Hermann, T. Koisk, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). find our card
M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv . find our card
D. Hewlett, A. Lacoste, L. Jones, I. Polosukhin, A. Fandrianto, J. Han, M. Kelcey, and D. Berthelot. 2016. Wikireading: A novel large-scale language understanding task over Wikipedia. In Association for Computational Linguistics (ACL). find
A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP. find our card
M. Hu, Y. Peng, and X. Qiu. 2017. Reinforced mnemonic reader for machine comprehension. arXiv . find
M. Wang, N. A. Smith, and T. Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for QA. In Empirical Methods in Natural Language Processing (EMNLP). find
H. Huang, C. Zhu, Y. Shen, and W. Chen. 2018. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. In International Conference on Learning Representations (ICLR). find
W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Association for Computational Linguistics (ACL). find
R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). find our card
D. Weissenborn, G. Wiese, and L. Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Computational Natural Language Learning (CoNLL). find
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). find our card
Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP). pages 2013–2018. find our card
G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 . find our card
W. Yih, M. Chang, C. Meek, and A. Pastusiak. 2013. Question answering using enhanced lexical semantic models. In Association for Computational Linguistics (ACL). find
O. Levy, M. Seo, E. Choi, and L. Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Computational Natural Language Learning (CoNLL). find
Y. Zhang, V. Zhong, D. Chen, G. Angeli, and C. D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Empirical Methods in Natural Language Processing (EMNLP). find
M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. bernardi, and R. Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Language Resources and Evaluation Conference (LREC). find