CoQA: A Conversational Question Answering Challenge
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q19-1016.pdf
Summary
Most reading comprehension benchmarks treat each question on its own. People, by contrast, gather information through chains of follow-up questions that depend on earlier turns. The authors built CoQA to test whether machines can keep up such a dialogue about a text. Pairs of crowdworkers on Amazon Mechanical Turk, connected through the ParlAI MTurk API, talked about a passage. One asked questions. The other highlighted a supporting span (the rationale) and then edited it into a short, natural free-form answer. Passages come from seven domains, and two are held out for out-of-domain testing. Every dev and test question received three extra answers to account for valid variation. The analysis shows that almost half the questions use anaphora that point back to the history, and many require pragmatic inference, so lexical matching alone does not work. Strong conversational and reading comprehension baselines were tested. The best scores 65.4 F1, compared with 88.8 for humans. CoQA serves as a standard testbed for context-dependent QA and is a useful template for building similar datasets in other languages, such as Ukrainian.
Key points
- Size: about 127k question–answer turns from roughly 8k conversations, averaging about 15 turns each. Answers are short, about 2.7 words on average.
- Domains: children's stories (MCTest), literature (Project Gutenberg), school English exams (RACE), CNN news and Wikipedia are in-domain. Reddit (Writing Prompts) and science (AI2 Science Questions) are out-of-domain and appear only in the test set.
- Answers are free-form, and each has a span rationale from the passage. Rationales are not available at test time, and unanswerable questions are labelled "unknown".
- Best system: 65.4 F1 against 88.8 F1 for humans, a gap of 23.4 points. A combined DrQA-plus-generation model also produces rationales and handles counting and multiple-choice questions.
- Compared with QuAC (Choi et al., 2018), where answers must be extractive spans and passages are Wikipedia articles about people only, CoQA has higher human agreement (88.8 vs 74.6 F1) and covers more domains.
Cited by
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- QuAC: Question Answering in Context
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Pre-Training with Whole Word Masking for Chinese BERT
- Revisiting Pre-Trained Models for Chinese Natural Language Processing
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- English Machine Reading Comprehension Datasets: A Survey
- ELI5: Long Form Question Answering
- HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark
- Conversational implicatures in English dialogue: Annotated dataset
- Evaluating Large Language Models: A Comprehensive Survey
- Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Inquisitive Question Generation for High Level Text Comprehension
- Natural Questions: A Benchmark for Question Answering Research
- DataComp-LM: In search of the next generation of training sets for language models
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning
- "I'd rather just go to bed": Understanding Indirect Answers
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
- A Comprehensive Overview of Large Language Models
- Language Models as Knowledge Bases?
- How Context Affects Language Models' Factual Predictions
- PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations
- Pre-trained Models for Natural Language Processing: A Survey
- A guide to the dataset explosion in QA, NLI, and commonsense reasoning
- Disentangling Indirect Answers to Yes-No Questions in Real Conversations
- Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- ASQA: Factoid Questions Meet Long-Form Answers
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- RedPajama: an Open Dataset for Training Large Language Models
- BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
- A Robustly Optimized BERT Pre-training Approach with Post-training
- VALUE: Understanding Dialect Disparity in NLU
- Multi-VALUE: A Framework for Cross-Dialectal English NLP
Cite
@misc{reddy2019coqa,
title = {{CoQA: A Conversational Question Answering Challenge}},
author = {Reddy, Siva and Chen, Danqi and Manning, Christopher D.},
year = {2019},
doi = {10.1162/tacl\_a\_00266},
eprint = {1808.07042},
archivePrefix = {arXiv},
url = {https://doi.org/10.1162/tacl_a_00266}
}References
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In International Conference on Computer Vision (ICCV), pages 2425–2433. Santiago, Chile. find