❓ Question answering

About 18 minutes to read · every paper named here has its own page in this library

What it is

A question answering system takes a question such as "When was Mozart born?" and returns "1756". It does not return a list of documents to read. This is what separates QA from search. A search engine ranks pages, while a QA system commits to an answer and, ideally, points to the evidence behind it. QA is one of the oldest goals of natural language processing. It also became one of its most heavily benchmarked tasks, because a question with a known answer is an easy way to probe whether a machine has understood a text.

The field is usually divided by where the answer comes from and what form it takes.

• Extractive reading comprehension. The system gets a question and a passage, and the answer is a contiguous span of that passage. SQuAD made this format standard (Rajpurkar et al. 2016).
• Open-domain QA. No passage is given. The system must first find relevant text in a large corpus such as Wikipedia, then read it (Karpukhin et al. 2020; Lee et al. 2019).
• Knowledge-base question answering (KBQA). The answer is looked up in a structured knowledge graph like Freebase, often by translating the question into an executable query (Yih et al. 2015).
• Multi-hop QA. Answering requires combining facts from two or more documents, as in "Which river flows through the birthplace of X?" (Yang et al. 2018).
• Visual QA. The question is about an image, a video or a chart (Goyal et al. 2016; Lei et al. 2018; Masry et al. 2022).
• Conversational QA. Questions arrive in a dialogue, and later ones depend on earlier turns (Reddy et al. 2019; Choi et al. 2018).

Answers also differ in form. They can be a span, a yes/no verdict, a choice among options, a number computed from the text, or a free-form paragraph. Long-form QA, which asks for explanatory answers to "why" and "how" questions, sits close to text generation (Fan et al. 2019).

How it works

The retriever–reader pipeline

Most modern open-domain systems split the work into two stages. A retriever narrows a corpus of millions of passages down to a handful of candidates. A reader then examines those candidates and produces the answer. The design echoes the TREC question answering evaluations of the late 1990s and 2000s. Chen and colleagues (2017) recast it in neural form in DrQA, described in Reading Wikipedia to Answer Open-Domain Questions, with Wikipedia as the knowledge source (Clark & Gardner 2017).

For years the retriever was a sparse keyword method: term frequency–inverse document frequency (TF-IDF) or BM25, term-weighting schemes from information retrieval. Dense retrieval replaces keyword overlap with learned vectors. Two encoders map the question and each passage into the same vector space, and relevance is measured by the inner product:

sim(q, p) = E_Q(q) · E_P(p)

Here q is the question and p a passage. E_Q and E_P are the question and passage encoders, each a neural network that outputs a fixed-length vector. The dot "·" is the dot product. A higher score means the passage is judged more likely to contain the answer.

Because passage vectors do not depend on the question, they can be computed once, stored, and searched with an approximate nearest-neighbour index. In the Dense Passage Retriever, both encoders are BERT-base models, and the representation is the 768-dimensional vector of the first token (Karpukhin et al. 2020). The earlier ORQA system used 128-dimensional encodings of more than 13 million Wikipedia blocks (Lee et al. 2019).

Training pushes the score of the correct passage above the scores of incorrect ones. A common objective is a softmax over one positive and several negatives:

L = − log [ exp(sim(q, p⁺)) / ( exp(sim(q, p⁺)) + Σⱼ exp(sim(q, pⱼ⁻)) ) ]

p⁺ is a passage that contains the answer, the pⱼ⁻ are negative passages, and the sum runs over all negatives. L is the loss to be minimised.

DPR's central finding was that cheap negatives are enough. It reused the gold passages of the other questions in the same mini-batch and added one BM25 hard negative per question. No extra pretraining was needed (Karpukhin et al. 2020). On Natural Questions, top-5 retrieval accuracy rose from 42.9% with BM25 to 65.2% with DPR (Karpukhin et al. 2020).

Span prediction

An extractive reader encodes the question and passage together, producing a vector hᵢ for every passage token i. Two learned vectors then score each token as a possible start or end of the answer:

P(start = i) = exp(wₛ · hᵢ) / Σⱼ exp(wₛ · hⱼ)
P(end = k)   = exp(wₑ · hₖ) / Σⱼ exp(wₑ · hⱼ)

wₛ and wₑ are trained parameter vectors, hᵢ is the contextual representation of token i, and the sums run over all tokens j of the passage. The predicted answer is the span (i, k) with i ≤ k, usually under a maximum length, that maximises the product of the two probabilities. Training minimises the negative log probability of the gold start and end positions (Wang et al. 2017).

Single-pass prediction can lock in an early mistake. The Dynamic Coattention Network addressed this by re-estimating start and end positions over several iterations (Xiong et al. 2016).

When a reader sees many passages, the raw scores from different passages are not comparable. Clark and Gardner fixed this with a shared-normalisation objective that applies the softmax jointly across all paragraphs from one document (Clark & Gardner 2017).

Generative readers and language models

A generative reader writes the answer token by token instead of pointing to it. Retrieval-augmented generation couples a retriever with a sequence-to-sequence language model for knowledge-intensive tasks (Lewis et al. 2020).

Fusion-in-Decoder takes a different route. It encodes each retrieved passage separately with the question, and lets a T5 decoder attend over all of the encodings at once. Cost therefore grows linearly with the number of passages, so the reader can take 100 of them (Izacard & Grave 2021). The same work contrasts this with closed-book generation, where a model answers from its weights alone. A 770M-parameter reader with BM25 retrieval outscored an 11B-parameter closed-book T5 on Natural Questions (Izacard & Grave 2021).

At the far end of this line, WebGPT fine-tuned GPT-3 to drive a text-only browser. The model learned to search, click and quote, and was trained from human demonstrations and preference comparisons (Nakano et al. 2021). These systems connect QA to retrieval-augmented generation and to large language models more broadly.

Measuring answers

Extractive QA is usually scored with two measures introduced with SQuAD (Rajpurkar et al. 2016). Exact match (EM) is 1 if the normalised prediction equals a reference answer and 0 otherwise. Normalisation typically lowercases the text and strips punctuation and articles. Token-level F1 treats the prediction and the reference as bags of tokens:

F1 = 2 · P · R / (P + R)

P (precision) is the share of predicted tokens that appear in the reference. R (recall) is the share of reference tokens that appear in the prediction. When several references exist, the maximum over them is taken. Both measures are averaged over questions.

Free-form answers are harder to score. MS MARCO reported ROUGE-L (Bajaj et al. 2016), and its free-form answers were also scored with bilingual evaluation understudy (BLEU) (Kwiatkowski et al. 2019). Long-form QA relies heavily on human preference judgements (Fan et al. 2019; Nakano et al. 2021).

History

Patterns and the TREC era

Early systems such as BASEBALL (Green and colleagues, 1961) and LUNAR (Woods, 1973) answered questions over small structured databases. They were forerunners of today's knowledge-base QA. Open-domain QA over text became a shared evaluation at TREC, where answer extraction often depended on surface patterns.

Ravichandran and Hovy learned such patterns automatically. They started from a few seed question–answer pairs, mined web sentences with suffix trees, and estimated each pattern's precision. For instance, a pattern made of the person's name followed by an opening parenthesis, the answer and a dash proved fully reliable for birth dates (Ravichandran & Hovy 2002). They found that answers mined from the web beat answers drawn from the TREC corpus, and credited the redundancy of the web (Ravichandran & Hovy 2002).

Answer sentence selection was benchmarked on WikiQA, built from real Bing queries. On the older QASent set, derived from TREC-QA, lexical word-matching models such as LCLR did well. On WikiQA, sentence-level neural models did better (Yang et al. 2015).

Large datasets and neural readers

The shift to deep learning came with large datasets. Hermann and colleagues (2015) turned news summaries into cloze questions. A careful re-examination showed that a simple attentive model reached 73.6% on CNN and 76.6% on Daily Mail. The same study found that about a quarter of the questions were damaged by coreference errors or anonymisation, so the benchmark said little about multi-sentence reasoning (Chen et al. 2016).

SQuAD, released in 2016, offered 107,785 crowd-written questions over Wikipedia paragraphs with span answers. Its feature-based baseline reached 51.0% F1 against 86.8% for humans (Rajpurkar et al. 2016). A rapid sequence of neural readers followed, each built from recurrent encoders and new forms of attention between question and passage:

• BiDAF (Seo et al. 2016);
• the Dynamic Coattention Network (Xiong et al. 2016);
• gated self-matching networks (Wang et al. 2017).

Intra-attention inside recurrent readers had been explored for machine reading just before (Cheng et al. 2016). QANet, from Carnegie Mellon University and Google Brain, then removed recurrence altogether. It paired convolutions with self-attention in the style of the Transformer and trained several times faster at similar accuracy (Yu et al. 2018). Pretrained encoders, notably BERT (Devlin and colleagues, 2018), soon made a fine-tuned span head on a large pretrained model the default extractive reader.

From reading to retrieval and generation

After reading came finding. ORQA trained retriever and reader together from question–answer pairs alone, treating the evidence passage as a latent variable (Lee et al. 2019). It pretrained the retriever with an Inverse Cloze Task, in which the model is given one sentence and must identify the passage that originally contained it (Lee et al. 2019). DPR showed that ordinary supervised training of a dual encoder was enough (Karpukhin et al. 2020). RocketQA improved that training with cross-batch negatives, denoised hard negatives and pseudo-labelled data (Qu et al. 2021).

Methods

Extractive reading comprehension

The neural readers of 2016–2018 shared one recipe:

1. Embed words, often with GloVe word embeddings plus character-level features.
2. Encode question and passage in context.
3. Fuse the two with attention.
4. Predict a span.

BiDAF computed one similarity matrix and used it for attention in both directions. Its attention was computed at every passage position rather than squeezed into a single vector (Seo et al. 2016). Gated self-matching added a relevance gate and let the passage attend over itself to gather distant evidence (Wang et al. 2017). These readers became standard baselines on many later datasets, including TriviaQA (Joshi et al. 2017), DROP (Dua et al. 2019) and QuAC (Choi et al. 2018).

Dense, sparse and hybrid retrieval

Dense retrievers capture paraphrase that keywords miss, but they have blind spots. ORQA beat BM25 on datasets whose questions came from people who actually needed the answer. BM25 stayed stronger where the question writers already knew the answer, as in SQuAD and TriviaQA (Lee et al. 2019). The same work noted that compact vectors struggle with very specific concepts and suggested hybrid sparse–dense retrieval (Lee et al. 2019).

Hard negatives are a delicate part of training. Manual inspection of MS MARCO found that about 70% of top-ranked but unlabelled passages were in fact relevant, so many "negatives" are false ones (Qu et al. 2021).

Generation with retrieval

Generative readers can combine evidence from many passages and can produce answers that are not literal spans. Fusion-in-Decoder kept improving as passages were added. Going from 10 to 100 passages added about 6 points on TriviaQA and 3.5 on Natural Questions, whereas extractive readers usually stopped gaining at 10–20 passages (Izacard & Grave 2021). For long-form answers, the evidence pipeline itself is a bottleneck. In ELI5, fewer than 9% of the best-matching sentences came from the top 20 retrieved passages (Fan et al. 2019).

Knowledge-base QA and semantic parsing

KBQA usually maps a question onto a query over the graph, which links it to semantic parsing. STAGG built a query graph in stages: it chose a topic entity, then a chain of relations, then constraints and aggregations. Because the knowledge base guided every step, the search space shrank. The system reached 52.5% F1 on WebQuestions (Yih et al. 2015).

Labelling questions with full SPARQL parses, rather than answers only, raised average F1 from 66.8% to 71.7%. Per question, it also took experts less time than finding the answers by hand (Yih et al. 2016).

Real knowledge graphs are incomplete. EmbedKGQA scored candidate answers with ComplEx embeddings, which represent the graph's entities and relations as complex-valued vectors. This let it recover answers even when the direct link had been deleted (Saxena et al. 2020). Neural module networks took a compositional route. They assembled a network per question from small modules such as find, relate and exists, and worked over both images and a geographic database (Andreas et al. 2016).

Visual QA

Visual readers fuse an image representation with the question. Stacked attention networks let the question vector attend over a 14×14 grid of image regions in successive steps, narrowing focus each time (Yang et al. 2015). Multimodal Compact Bilinear pooling approximated the full outer product of image and text vectors cheaply (Fukui et al. 2016). It hashed each vector into a larger space with Count Sketch, a random projection. It then combined the two sketches by multiplying them element-wise after a fast Fourier transform (FFT), which is equivalent to convolving them (Fukui et al. 2016). Grounding answers to boxes in the image made it possible to inspect where a model looks (Zhu et al. 2015). This line of work belongs to multimodal learning.

QA as a general interface and as a data source

Questions can also serve as a format for other tasks. Event extraction was recast as extractive QA: one model is queried for triggers and another is asked role-specific questions. This removed the need for entity annotation and allowed zero-shot extraction of unseen roles (Du & Cardie 2020). It ties QA to information extraction.

In the opposite direction, neural question generation trained a sequence-to-sequence model on SQuAD sentence–question pairs. Its questions beat a rule-based system in human judgements (Du et al. 2017). Generated questions can then be used to build new datasets (Du et al. 2017).

Datasets and benchmarks

Benchmarks have driven the field, and each new one was usually built to expose what the previous one let models skip. The main families are summarised below. Sizes are as reported by the creators, and the unit is given in each row.

Reading comprehension

SQuAD's paragraphs come from 536 high-PageRank Wikipedia articles. Its analysis showed that the difficulty lay mostly in finding the exact span, not the right sentence (Rajpurkar et al. 2016).

TriviaQA reversed the order of construction. Trivia enthusiasts wrote the questions first, and evidence documents were gathered afterwards. As a result, questions and evidence share less wording (Joshi et al. 2017).

RACE drew multiple-choice questions from English exams for Chinese schoolchildren. Its options are free-form sentences rather than spans (Lai et al. 2017).

Several datasets targeted reasoning that span matching cannot do:

• MultiRC required evidence from several sentences, and a question could have any number of correct options (Khashabi et al. 2018).
• DROP demanded counting, sorting and arithmetic over paragraphs, and was collected adversarially against BiDAF (Dua et al. 2019).
• NarrativeQA posed questions over whole stories (Kočiský et al. 2018).

Open-domain and real information needs

MS MARCO paired anonymised Bing queries with retrieved web passages and human-written answers. It became a standard for neural ranking as well as for reading (Bajaj et al. 2016). Natural Questions used real Google queries of eight or more words, each paired with a Wikipedia page from the top results. Annotators marked a long answer, a short answer, or NULL (Kwiatkowski et al. 2019). In this original form a system reads the given page, and open versions without the page were used later to evaluate retrievers (Lee et al. 2019). Only about 49% of its examples have a non-NULL long answer, and repeated annotation showed that many real questions have several acceptable answers (Kwiatkowski et al. 2019).

Multi-hop, knowledge and commonsense

HotpotQA linked Wikipedia paragraphs through bridge entities: an entity described in one paragraph and mentioned in the other, through which the reasoning must pass. It also built comparison pairs. Workers marked the sentences each answer depends on, so systems can be scored on evidence as well as on answers (Yang et al. 2018). 2WikiMultiHopQA added Wikidata triples that trace the reasoning path, and removed templates known to yield single-hop shortcuts (Ho et al. 2020). WebQuestionsSP supplies executable SPARQL parses for KBQA (Yih et al. 2016).

Several benchmarks test knowledge that is not stated in any passage, and they form a large part of research on reasoning:

• CommonsenseQA built five-way questions from ConceptNet relations (Talmor et al. 2019).
• ARC collected grade-school science questions (Clark et al. 2018).
• OpenBookQA pairs questions with a small book of science facts (Mihaylov et al. 2018).

Conversational, multilingual, visual and domain QA

CoQA has free-form answers backed by span rationales, and nearly half its questions contain references back to earlier turns (Reddy et al. 2019). QuAC hid the evidence from the asker, which produced more exploratory questions. About 20% of its questions have no answer (Choi et al. 2018). Both link QA to dialogue systems.

For multilingual NLP, MLQA aligned Wikipedia sentences across seven languages and translated the questions by hand (Lewis et al. 2019). TyDi QA instead collected native questions in 11 typologically diverse languages, with no translation at any stage (Clark et al. 2020).

Visual QA moved through several datasets:

• VQA v2.0, which pairs nearly every question with two similar images that need different answers (Goyal et al. 2016);
• Visual7W, with answers grounded to image regions (Zhu et al. 2015);
• GQA, with compositional questions generated from scene graphs (Hudson & Manning 2019);
• TVQA, on television clips with subtitles (Lei et al. 2018);
• ChartQA, on real charts (Masry et al. 2022).

PubMedQA asks yes/no/maybe questions about biomedical abstracts (Jin et al. 2019).

Unanswerable questions

A system that must always return a span will guess even when the text is silent. WikiQA kept questions with no correct sentence and defined answer triggering, the decision of whether any candidate answers at all. Most of its questions have no correct candidate (Yang et al. 2015).

SQuAD 2.0 made abstention central. Crowdworkers wrote on-topic unanswerable questions, and each paragraph contained a plausible decoy span of the expected answer type (Rajpurkar et al. 2018). Baselines predicted a no-answer probability and abstained above a threshold tuned on the dev set (Rajpurkar et al. 2018). Automatically generated negatives proved much easier than these written ones (Rajpurkar et al. 2018). Natural Questions, TyDi QA and MS MARCO all contain unanswerable items that arise naturally from real queries (Kwiatkowski et al. 2019; Clark et al. 2020; Bajaj et al. 2016).

For how such numbers are produced and read, see evaluation and benchmarks.

People and institutions

Much of the modern field traces to a few groups that built both datasets and models.

At Stanford, Percy Liang and his co-authors created SQuAD and its unanswerable successor (Rajpurkar et al. 2016; Rajpurkar et al. 2018). Together with Robin Jia, Liang also built adversarial tests showing that reading systems can be distracted (Jia & Liang 2017).

Christopher D. Manning, at Stanford University, co-authored three benchmarks (Yang et al. 2018; Hudson & Manning 2019; Reddy et al. 2019):

• HotpotQA, for multi-hop QA;
• GQA, for visual reasoning;
• CoQA, for conversational QA.

Danqi Chen, now at Princeton University, examined the CNN/Daily Mail task (Chen et al. 2016) and co-authored CoQA (Reddy et al. 2019) and DPR (Karpukhin et al. 2020).

The University of Washington is another centre. Luke Zettlemoyer and Eunsol Choi built TriviaQA (Joshi et al. 2017). Choi also led QuAC (Choi et al. 2018) and contributed to TyDi QA (Clark et al. 2020). Hannaneh Hajishirzi, at the University of Washington and the Allen Institute for Artificial Intelligence, co-authored BiDAF (Seo et al. 2016). Yejin Choi, now at Stanford University, and Mohit Iyyer were co-authors of QuAC (Choi et al. 2018).

Researchers at Google built several of the main benchmarks of real information needs. Tom Kwiatkowski and Michael Collins were among the authors of Natural Questions (Kwiatkowski et al. 2019) and TyDi QA (Clark et al. 2020). Ming-Wei Chang co-developed ORQA (Lee et al. 2019) and, earlier, the query-graph KBQA work (Yih et al. 2015). MS MARCO came from Microsoft (Bajaj et al. 2016).

Researchers at Meta (then Facebook AI Research) produced DPR, retrieval-augmented generation and MLQA. Patrick Lewis, now at Cohere, co-authored MLQA (Lewis et al. 2019), DPR (Karpukhin et al. 2020) and retrieval-augmented generation (Lewis et al. 2020). Sewon Min was also a DPR author (Karpukhin et al. 2020). Wen-tau Yih, now at Meta's FAIR lab and earlier at Microsoft Research, has worked across the whole range of the field:

• WikiQA, for answer sentence selection (Yang et al. 2015);
• the staged query-graph approach to KBQA (Yih et al. 2015);
• WebQuestionsSP (Yih et al. 2016);
• QuAC (Choi et al. 2018);
• DPR (Karpukhin et al. 2020).

Others extended QA into new forms:

• Matt Gardner co-authored DROP (Dua et al. 2019) and multi-paragraph reading (Clark & Gardner 2017).
• Jonathan Berant, now at Google and Tel Aviv University, co-authored CommonsenseQA (Talmor et al. 2019).
• Claire Cardie, at Cornell University, worked on neural question generation (Du et al. 2017) and QA-style event extraction (Du & Cardie 2020).
• Mohit Bansal, at the University of North Carolina at Chapel Hill, co-authored TVQA (Lei et al. 2018).

Among institutions, Google, Carnegie Mellon University, the Allen Institute for Artificial Intelligence, Stanford University, the University of Washington, Microsoft and Meta are prominent research homes for the field.

Open questions

Shortcuts and robustness

High scores have repeatedly turned out to rest on surface cues. Adversarially inserted sentences exposed how fragile SQuAD-trained readers are (Jia & Liang 2017). In the original VQA data, answering "yes" to every "Do you see a…" question scored 87%. Every model tested lost accuracy once images were balanced (Goyal et al. 2016). In Visual7W, a question-only model beat humans who could not see the image. It did so by learning answer priors, that is, which answers tend to go with which questions in the training data (Zhu et al. 2015).

Whether a benchmark truly needs the skill it names, whether careful reading, multi-hop reasoning or looking at the image, remains a design problem for each new dataset (Jia & Liang 2017; Goyal et al. 2016; Ho et al. 2020). Detecting which cues a trained model actually relies on is a goal it shares with interpretability research.

Reasoning beyond spans

Discrete reasoning is still hard. On DROP, BERT reached 32.7% F1 against 96.4% for expert humans, and a numerically aware model reached 47.0% (Dua et al. 2019). CommonsenseQA's learning curve suggested that even 100K training examples would bring BERT-large only to about 75% accuracy (Talmor et al. 2019). In ChartQA, errors in nested operations and in extracting data from the chart kept accuracy low on human-written questions (Masry et al. 2022). Biomedical reasoning over quantitative findings also left a gap: 68.1% for the best system versus 78.0% for a single human on PubMedQA (Jin et al. 2019).

Knowing when not to answer

Abstention is far from solved. On SQuAD 2.0, the best tested system trailed humans by 23.2 F1 points, and a system that always abstains already scored 48.9 F1 (Rajpurkar et al. 2018). With large generative models the problem changes shape. A fluent answer may be wrong, and WebGPT's citations were meant partly to make fact-checking easier (Nakano et al. 2021).

Retrieval quality and incomplete knowledge

Retrievers trained on incompletely labelled data learn from false negatives (Qu et al. 2021). Dense vectors can miss rare entities (Lee et al. 2019). Knowledge graphs lack links that questions need (Saxena et al. 2020). Combining sparse, dense and structured evidence in one system is still an open engineering and modelling problem.

Evaluation of free-form and ambiguous answers

Exact match punishes correct paraphrases, and F1 rewards partial overlap that may be wrong. Natural Questions showed that real questions are often ambiguous or have several valid answers, which forced multi-reference metrics and a lower human ceiling (Kwiatkowski et al. 2019). For long-form answers, automatic metrics track quality poorly. In ELI5, human raters preferred the gold answer over every model in at least 85.5% of comparisons (Fan et al. 2019). Truthfulness is a separate axis from fluency. On the adversarial TruthfulQA benchmark, 75% of WebGPT's answers were true, but only 54% were both true and informative (Nakano et al. 2021).

Languages beyond English

Cross-lingual transfer still drops sharply away from English. With XLM on MLQA, F1 fell from 74.9 in English to 48.8 in Hindi (Lewis et al. 2019). On TyDi QA, a SQuAD-trained model transferred zero-shot reached 56.4 F1 on the gold-passage task, against 79.0 after training on TyDi QA itself (Clark et al. 2020). Building native, information-seeking data for under-served languages remains a priority shared with work on low-resource languages.

The library in figures