SQuAD: 100,000+ Questions for Machine Comprehension of Text
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1264.pdf
Summary
Machine reading comprehension lacked a benchmark that was both large and natural. Carefully curated sets such as MCTest are too small for data-hungry models. Very large cloze-style corpora such as CNN/Daily Mail and CBT are semi-synthetic. To fill this gap, the authors built SQuAD. Crowdworkers read paragraphs from 536 English Wikipedia articles, wrote questions about them and highlighted each answer as a span of the paragraph. No answer options are provided, so a system has to choose the correct span from every possible candidate in the passage. Span answers are still simple to score automatically. The authors use dependency and constituency parses to describe how hard the questions are, for example by measuring how far a question's syntax differs from the sentence that holds its answer. A logistic regression model with many features reaches 51.0% F1. That is well above a sliding-window baseline of about 20% but far below human agreement of 86.8%. The resource is freely available and gives researchers a demanding benchmark. Neural models had already narrowed much of the gap soon after its release.
Key points
- Data: 107,785 question–answer pairs drawn from 23,215 paragraphs. The paragraphs come from 536 articles sampled from the 10,000 English Wikipedia pages with the highest PageRank. Articles are split 80/10/10 into train, dev and test sets. Workers used the Daemo platform on Amazon Mechanical Turk with copy-paste disabled, and every dev/test question received at least two more answers.
- Results (v1.0 test): logistic regression scores 40.4% EM and 51.0% F1. The sliding window scores 12.5% EM and 19.7% F1, and humans score 77.0% EM and 86.8% F1.
- Ablation: lexicalized features and dependency tree path features matter most. Removing both lowers dev F1 to 35.8%. The model picks the correct sentence 79.3% of the time, so most of the difficulty is in finding the exact span.
- Where the model struggles: it does best on dates and numbers and worst on non-entity answers, which make up about 47.6% of the data. It also gets worse as syntactic divergence increases, while human scores stay roughly constant.
- Follow-up: Wang and Jiang (2016) reached 70.3% F1 on v1.1, which more than halves the gap to human performance. The code and data are published on CodaLab.
Cited by
- AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level
- Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- MEGA: Multilingual Evaluation of Generative AI
- Yi: Open Foundation Models by 01.AI
- Detecting Language Model Attacks with Perplexity
- AraBERT: Transformer-based Model for Arabic Language Understanding
- On the Cross-lingual Transferability of Monolingual Representations
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
- STARC: Structured Annotations for Reading Comprehension
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- On the Opportunities and Risks of Foundation Models
- LEGAL-BERT: The Muppets straight out of Law School
- LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
- Reading Wikipedia to Answer Open-Domain Questions
- PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
- QuAC: Question Answering in Context
- Rethinking embedding coupling in pre-trained language models
- Simple and Effective Multi-Paragraph Reading Comprehension
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Transformers as Soft Reasoners over Language
- Pre-Training with Whole Word Masking for Chinese BERT
- Revisiting Pre-Trained Models for Chinese Natural Language Processing
- Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Learning to Ask: Neural Question Generation for Reading Comprehension
- Event Extraction by Answering (Almost) Natural Questions
- GLM: General Language Model Pretraining with Autoregressive Blank Infilling
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
- English Machine Reading Comprehension Datasets: A Survey
- QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
- ELI5: Long Form Question Answering
- Reducing Transformer Depth on Demand with Structured Dropout
- HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark
- Multilingual Sequence-to-Sequence Models for Hebrew NLP
- Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- Bias and Fairness in Large Language Models: A Survey
- Making Pre-trained Language Models Better Few-shot Learners
- Enabling Large Language Models to Generate Text with Citations
- Retrieval-Augmented Generation for Large Language Models: A Survey
- AllenNLP: A Deep Semantic Natural Language Processing Platform
- Injecting Numerical Reasoning Skills into Language Models
- Parameter-Efficient Transfer Learning with Diff Pruning
- Evaluating Large Language Models: A Comprehensive Survey
- DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- Bridging Anaphora Resolution as Question Answering
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
- Large Language Models Can Self-Improve
- TrustLLM: Trustworthiness in Large Language Models
- Editing Models with Task Arithmetic
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
- Adversarial Examples for Evaluating Reading Comprehension Systems
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
- PubMedQA: A Dataset for Biomedical Research Question Answering
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- SpanBERT: Improving Pre-training by Representing and Predicting Spans
- The State and Fate of Linguistic Diversity and Inclusion in the NLP World
- Evaluating Open-Domain Question Answering in the Era of Large Language Models
- Large Language Models Struggle to Learn Long-Tail Knowledge
- Dense Passage Retrieval for Open-Domain Question Answering
- RealTime QA: What's the Answer Right Now?
- Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
- How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
- UNIFIEDQA: Crossing Format Boundaries with a Single QA System
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- Inquisitive Question Generation for High Level Text Comprehension
- The NarrativeQA Reading Comprehension Challenge
- Revealing the Dark Secrets of BERT
- Reformulating Unsupervised Style Transfer as Paraphrase Generation
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Natural Questions: A Benchmark for Question Answering Research
- Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Cite
@inproceedings{rajpurkar2016squad,
title = {{SQuAD: 100,000+ Questions for Machine Comprehension of Text}},
author = {Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy},
booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
year = {2016},
doi = {10.18653/v1/d16-1264},
eprint = {1606.05250},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d16-1264}
}References
- J. Berant, V. Srikumar, P. Chen, A. V. Linden, B. Harding, B. Huang, P. Clark, and C. D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). find
- E. Brill, S. Dumais, and M. Banko. 2002. An analysis of the AskMSR question-answering system. In Association for Computational Linguistics (ACL), pages 257– 264. find
- D. Chen, J. Bolton, and C. D. Manning. 2016. A thorough examination of the CNN / Daily Mail reading comprehension task. In Association for Computational Linguistics (ACL). find our card
- P. Clark and O. Etzioni. 2016. My computer is an honor student but how intelligent is it? standardized tests as a measure of AI. AI Magazine, 37(1):5–12. find
- J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), pages 248–255. find
- D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty. 2013. Building Watson: An overview of the DeepQA project. AI Magazine, 31(3):59–79. find
- S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Raj-pal, J. Regino, et al. 2015. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, pages 101–102. K. M. Hermann,… find our card
- F. Hill, A. Bordes, S. Chopra, and J. Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). find our card
- L. Hirschman, M. Light, E. Breck, and J. D. Burger. 1999. Deep read: A reading comprehension system. In Association for Computational Linguistics (ACL), pages 325–332. find
- M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Empirical Methods in Natural Language Processing (EMNLP), pages 523–533. find our card
- N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay. 2014. Learning to automatically solve algebra word problems. In Association for Computational Linguistics (ACL). find
- M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19:313–330. find our card
- K. Narasimhan and R. Barzilay. 2015. Machine comprehension with discourse relations. In Association for Computational Linguistics (ACL). find
- H. T. Ng, L. H. Teo, and J. L. P. Kwan. 2000. A machine learning approach to answering questions for reading comprehension tests. In Joint SIGDAT conference on empirical methods in natural language processing and very large corpora - Volume 13, pages 124–132. find
- D. Ravichandran and E. Hovy. 2002. Learning surface text patterns for a question answering system. In Association for Computational Linguistics (ACL), pages 41–47. find
- M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193– 203. find our card
- E. Riloff and M. Thelen. 2000. A rule-based question answering system for reading comprehension tests. In ANLP/NAACL Workshop on reading comprehension tests as evaluation for computer-based language understanding sytems - Volume 6, pages 13–19. find
- M. Sachan, A. Dubey, E. P. Xing, and M. Richardson. 2015. Learning answer-entailing structures for machine comprehension. In Association for Computational Linguistics (ACL). find
- D. Shen and D. Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction. In International Conference on Computational Linguistics and Association for Computational Linguistics (COL-ING/ACL), pages 889–896. find
- M. Shirakawa, T. Hara, and S. Nishio. 2015. N-gram idf: A global term weighting scheme based on information distance. In World Wide Web (WWW), pages 960–970. find
- H. Sun, N. Duan, Y. Duan, and M. Zhou. 2013. Answer extraction from passage graph for question answering. In International Joint Conference on Artificial Intelligence (IJCAI). find
- E. M. Voorhees and D. M. Tice. 2000. Building a question answering test collection. In ACM Special Interest Group on Information Retreival (SIGIR), pages 200– 207. find
- Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. CoRR, abs/1608.07905. find our card
- H. Wang, M. Bansal, K. Gimpel, and D. McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Association for Computational Linguistics (ACL). find
- J. Weston, A. Bordes, S. Chopra, and T. Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv. find our card
- Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP), pages 2013–2018. find our card