📝 Text summarization

About 18 minutes to read · every paper named here has its own page in this library

What it is

A summary answers a practical question: what does a reader need to know from this text without reading all of it? A program that does this automatically is a text summarizer. Summarization is one of the oldest applications of natural language processing. It sits where natural language understanding meets natural language generation. The system must decide what matters, and then it must say it in fewer words. Research papers on the subject are gathered under the summarization topic.

Extractive and abstractive

The basic split is between two strategies. An extractive summarizer selects sentences, or smaller units such as clauses, from the source and joins them. An abstractive summarizer generates new text that may paraphrase, merge or generalize what the source says.

A short example shows the difference. Take this three-sentence news item:

1. "The city council met on Tuesday evening to discuss the budget."
2. "After four hours of debate, members voted 7 to 2 to close the Elm Street library."
3. "Residents had protested the plan for weeks."

An extractive summary limited to one sentence would probably pick sentence 2 unchanged. An abstractive summary might read: "Despite weeks of protest, the council voted to close the Elm Street library." That sentence combines content from sentences 2 and 3. It also contains words that appear in neither.

The abstractive version is more compact. It also carries a risk that an extractive summary avoids. "Despite" adds a concessive link, setting the protest against the vote, that the source only implies. Had the system written "unanimously", it would have stated something false. This trade-off between fluency and faithfulness shapes the whole recent history of the field.

Kinds of summarization tasks

Tasks also differ in their input and in what the reader wants:

• Single-document summarization condenses one text, typically a news article or a scientific paper.
• Multi-document summarization condenses a cluster of related texts, for example many reports of the same event. Here redundancy across sources and the order of events become central. Goldstein and colleagues argued that this is not just single-document summarization run several times. Related articles share background, later reports can override earlier ones, and the compression needed is far stronger (Goldstein et al. 2000).
• Query-focused summarization produces a summary that answers a given question or topic. This links it to question answering and to information retrieval.
• Dialogue and meeting summarization condenses conversations with several speakers, topic shifts and informal language. It relates the field to dialogue systems.
• Opinion summarization condenses many reviews into the points they share (Ganesan et al. 2010; Titov & McDonald 2008).

The same ideas also reach beyond prose. Source code summarization, for instance, generates short descriptions of functions from their code (Ahmad et al. 2020).

How it works

Almost every summarizer, old or new, has to solve three subproblems:

• Content selection: deciding which information is salient.
• Redundancy control: avoiding saying the same thing twice.
• Realization: producing readable output within a length budget.

Classical systems handled these in separate stages. Neural systems often learn them jointly. Some still keep a separate module, such as a content selector that runs before generation (Gehrmann et al. 2018).

Selection and redundancy

In extractive systems, each sentence receives a salience score. The final summary is then chosen by an algorithm that balances high scores against overlap with sentences already chosen.

The best-known balancing rule is Maximal Marginal Relevance, introduced by Carbonell and Goldstein (1998):

MMR = argmax over s ∈ R \ S  [ λ · Sim₁(s, Q) − (1 − λ) · max over t ∈ S  Sim₂(s, t) ]

The symbols mean the following:

• R is the set of candidate sentences, and S is the set already selected.
• Q is the query or the document centroid.
• Sim₁ measures relevance and Sim₂ measures similarity between sentences.
• λ, between 0 and 1, sets the balance between relevance and novelty.

Lin and Bilmes later showed that MMR and concept-coverage methods are special cases of submodular optimization (Lin & Bilmes 2011). Submodular functions have diminishing returns. When such a function is also monotone, a greedy procedure is guaranteed to reach at least (1 − 1/e) ≈ 0.632 of the optimal value under a cardinality limit. Under a word budget, which is a knapsack constraint, a cost-scaled greedy variant guarantees a (1 − 1/√e) factor. MMR itself is submodular but not monotone, so that guarantee does not carry over to it (Lin & Bilmes 2011).

Gillick and Favre cast selection as an integer linear program (ILP) instead. A summary earns credit once for each weighted bigram concept it covers. Redundancy is therefore controlled across the whole summary instead of between pairs of sentences (Gillick & Favre 2009).

Generation

Abstractive systems treat summarization as conditional text generation. An encoder reads the source. A decoder then produces the summary token by token, attending to the source at each step. Most modern decoders use Transformer attention. Beam search and penalties on length and repetition shape the output.

A copy mechanism lets the decoder reproduce rare words and names directly from the input. This makes outputs more faithful, but also more extractive. Gehrmann and colleagues measured the effect. Novel words made up 14.8% of reference summaries. They made up only 2.2% of pointer-generator output, and about 0.5% once a content-selection mask was added (Gehrmann et al. 2018).

History

Early statistical summarization

The idea that frequent content words signal a document's topic goes back to Luhn (1958). The idea that sentence position, cue phrases and title words help goes back to Edmundson (1969). Both ideas survived for decades as features and baselines.

By the late 1990s, researchers looked for structure that was richer than word counts but cheaper than full semantic interpretation:

• Barzilay and Elhadad built lexical chains from WordNet relations to trace how a topic develops through a text. They extracted the sentences that represent the strongest chains (Barzilay & Elhadad 1997). Their chain-building algorithm differed from the greedy method of Hirst and St-Onge, which commits early to one word sense. Sense ambiguity is the problem studied in word sense disambiguation.
• Lin and Hovy learned "topic signatures", weighted term lists tied to a topic, from labelled document collections. These beat a position baseline in 34 of 40 test points: four topics, each scored at ten extract lengths (Lin & Hovy 2000).

Multi-document news and the DUC era

The Document Understanding Conferences (DUC) made multi-document news summarization a shared task. So did the later Text Analysis Conference (TAC) evaluations. Both were run by the National Institute of Standards and Technology (NIST).

Columbia researchers pursued generation rather than extraction. One line of work intersected dependency trees from sentences reporting the same fact. It then realized the shared content as new text (Barzilay et al. 1999). Later work turned this into sentence fusion inside the Newsblaster news browser (Barzilay & McKeown 2005).

The open MEAD platform combined centroid, position and graph-centrality features with rerankers for redundancy (Radev et al. 2004). In 2004 three foundations appeared:

• ROUGE, an automatic evaluation package (Lin 2004).
• The Pyramid method for human content evaluation (Nenkova & Passonneau 2004).
• Graph-based ranking with TextRank and with LexRank, which was published in the Journal of Artificial Intelligence Research that year (Mihalcea & Tarau 2004; Erkan & Radev 2011).

Probabilistic content models followed. One approach used a hidden Markov model over domain topics, which supported both sentence ordering and extraction (Barzilay & Lee 2004). Another used topic modeling in the style of latent Dirichlet allocation (LDA). It chose sentences whose word distribution best matched the document set (Haghighi & Vanderwende 2009).

The neural turn

From 2015 the field moved quickly to neural sequence models:

• Rush, Chopra and Weston trained an attention-based model to rewrite a sentence as a headline (Rush et al. 2015). Recurrent decoders soon followed (Chopra et al. 2016).
• Nallapati and colleagues extended sequence-to-sequence recurrent neural networks to longer news summaries (Nallapati et al. 2016).
• Neural extractors learned the sentence features that had previously been hand-crafted (Cheng & Lapata 2016; Nallapati et al. 2016).
• See, Liu and Manning combined a pointer that copies source words with a generator that produces vocabulary words (See et al. 2017). The design became the reference abstractive baseline for several years.
• Two further approaches pushed scores higher. One was reinforcement learning with ROUGE rewards (Paulus et al. 2017). The other was hybrid extract-then-rewrite systems (Chen & Bansal 2018).

Pretraining and large language models

After 2018, pretrained encoders and encoder-decoders became the default approach (Liu & Lapata 2019; Zhang et al. 2019). At the same time, a critical literature showed that ROUGE gains were masking factual errors (Kryscinski et al. 2019; Maynez et al. 2020).

Instruction-tuned large language models then arrived. Their zero-shot and few-shot summaries began to rival human writing on news (Zhang et al. 2024). Adapted models also reached expert quality in some clinical tasks (Veen et al. 2023).

Methods

Frequency, position and topic statistics

Simple statistics remain strong in news. The Lead-3 baseline copies an article's first three sentences. Leading neural systems trained on CNN/DailyMail match it or beat it only narrowly. Their outputs rely heavily on the way news articles put key facts first (Kryscinski et al. 2019).

Frequency works for multi-document input because repetition across sources signals importance. On DUC data, a bigram's document frequency predicted how likely it was to appear in human summaries, with ρ = 0.95 for DUC 2007 (Gillick & Favre 2009). KLSum chooses sentences that keep the summary's word distribution as close as possible, in KL divergence, to that of the input. TopicSum and HierSum improved on it by learning background and sub-topic distributions (Haghighi & Vanderwende 2009).

Graph-based ranking: TextRank and LexRank

TextRank represents a text as a graph of sentences, or of words when extracting keywords. Edges are weighted by overlap, and nodes are ranked with a PageRank-style random walk (Mihalcea & Tarau 2004). LexRank uses the same intuition: salience is lexical centrality. A sentence that resembles many other sentences is likely to be central (Erkan & Radev 2011). MEAD included this score as a feature. It computed the score as eigenvector centrality on a thresholded graph of lexical similarity (Radev et al. 2004).

These unsupervised methods remain competitive baselines, especially on long inputs. On PubMed, LexRank's ROUGE-1 slightly exceeded that of a neural abstractive model (Cohan et al. 2018). Graph ideas also entered neural systems:

• Graph-based attention over encoded sentences (Tan et al. 2017).
• Heterogeneous graphs that link sentences through shared words (Wang et al. 2020).
• Graphs over discourse units and over chains found by coreference resolution (Xu et al. 2019).

The last of these draws on discourse analysis. Graph structure also appears outside classic summarization. Pang and Lee used minimum cuts to keep only subjective sentences before classifying sentiment. Their extracts kept about 60% of the words. With one classifier, accuracy rose from 82.8% to 86.4% (Pang & Lee 2004).

Neural extraction

Neural extractors frame the task as labelling each sentence as in or out of the summary. Examples include:

• A hierarchical reader with an attention-based extractor (Cheng & Lapata 2016).
• A bidirectional gated recurrent unit (GRU) classifier. It has separate, inspectable terms for content, salience, novelty and position (Nallapati et al. 2016).
• NeuSum, which rescores the remaining sentences after each choice. This conditioning reduced its reliance on the first three sentences. They supplied 58.64% of its picks, against 80.91% for an earlier extractor (Zhou et al. 2018).

Training labels are usually built greedily, by choosing sentences that maximize ROUGE against the reference. Such labels are noisy, which motivated two lines of work. HIBERT pre-trains a document-level encoder by masking whole sentences (Zhang et al. 2019). MatchSum scores whole candidate summaries rather than single sentences; it is described further under pretrained models (Zhong et al. 2020).

Sequence-to-sequence with attention and copying

The abstractive line began with generating headlines from the first sentences of Gigaword articles (Rush et al. 2015). A recurrent version, RAS, was more abstractive than its predecessor. It copied 74% of its summary words from the input, against 92% for ABS+ (Chopra et al. 2016).

For multi-sentence news summaries, models had to stop repeating themselves. The main remedies were:

• Pointer-generator networks with copying (See et al. 2017).
• Intra-attention combined with trigram blocking, trained with maximum likelihood, with policy gradient, or with a mix of the two. The highest ROUGE-1 on CNN/DailyMail, 41.16, came from the variant trained on the ROUGE reward alone. The mixed objective was favoured because it kept the output fluent (Paulus et al. 2017).
• Multiple communicating encoder agents for long inputs (Celikyilmaz et al. 2018).

Two-stage designs separate selection from writing. A bottom-up content mask decides which tokens the decoder may copy (Gehrmann et al. 2018). An extractor-plus-rewriter decodes 10–20 times faster than a whole-document pointer-generator (Chen & Bansal 2018).

Control tokens prepended to the input can steer length, entity focus or source style (Fan et al. 2018). For long, structured documents, hierarchical encoders weight word attention by section attention (Cohan et al. 2018).

Pretrained models: BERT-style encoders, BART and PEGASUS

Pretrained language models changed the baseline. Liu and Lapata fine-tuned pretrained encoders for both extractive and abstractive summarization (Liu & Lapata 2019). The denoising encoder-decoder BART, introduced by Lewis et al. (2020), became a common starting point for abstractive fine-tuning.

PEGASUS introduced an objective designed for summarization. Its gap-sentence generation hides the sentences judged most important, by their ROUGE with the rest of the document. The model must then regenerate them. The 568M-parameter model matched or beat the previous best ROUGE on all 12 benchmarks it was tested on. Fine-tuned on only 1000 examples, it surpassed the earlier state of the art on 6 datasets (Zhang et al. 2019).

MatchSum, mentioned above, ranks whole candidate extracts by their closeness to the document. It uses a Siamese BERT encoder: twin copies of BERT with shared weights embed the document and each candidate, so the two can be compared directly. It is trained contrastively, so that better candidates rank higher (Zhong et al. 2020). For multiple documents, Hierarchical Transformers exchange information between paragraphs through attention instead of flattening all sources into one sequence (Liu & Lapata 2019).

Large language models

Instruction-tuned large language models summarize from a prompt alone. This makes prompt engineering and few-shot learning part of the method.

In a human study of ten models on news, instruction tuning mattered more than size. An instruction-tuned GPT-3 with 350M parameters did about as well as the 175B base model. The best model's summaries were judged roughly equal to those of hired freelance writers (Zhang et al. 2024).

In clinical text, physicians evaluated GPT-4 with in-context examples. Its summaries were judged as good as the experts' in 45% of cases and better than theirs in 36% (Veen et al. 2023).

Evaluation: ROUGE

Human evaluation is expensive. A moderate DUC-scale campaign needs more than 3000 person-hours (Lin 2004). ROUGE replaces it with overlap counting against human references. It borrows from BLEU in machine translation. But where BLEU is based on precision, ROUGE is based on recall (Lin 2004). Its n-gram version is:

ROUGE-N = Σ_{R ∈ Refs} Σ_{g ∈ R} Count_match(g) / Σ_{R ∈ Refs} Σ_{g ∈ R} Count(g)

The symbols mean the following:

• Refs is the set of reference summaries, and R is one of them.
• g ranges over the n-grams of length N in a reference.
• Count(g) is how often g occurs in the reference.
• Count_match(g) is the largest number of times g co-occurs in the system summary and that reference.

ROUGE-1 and ROUGE-2 therefore measure unigram and bigram recall. Modern papers usually report an F-measure version.

ROUGE-L uses the longest common subsequence, so it rewards correct word order. Lin's example takes the reference "police killed the gunman" and two candidates, "police kill the gunman" and "the gunman kill police". Both candidates share the same bigrams with the reference. ROUGE-L still separates them, scoring the first 0.75 and the second 0.5. It ignores alternative subsequences and shorter shared fragments, however (Lin 2004). The package also includes ROUGE-W, which favours consecutive matches, and ROUGE-S/SU, which count skip-bigrams. On DUC data, ROUGE correlated highly with human judgments for single-document and very short summaries. The correlation was noticeably lower for multi-document summaries (Lin 2004).

The limits of ROUGE are well documented:

• It is purely lexical, so paraphrase is penalized.
• It does not check facts.
• It only weakly tracks human judgments of modern systems (Kryscinski et al. 2019).
• It depends on reference quality. Against the original XSum references, ROUGE-L scores moved opposite to human faithfulness ratings. Against freelance-written references, the relationship became positive (Zhang et al. 2024).
• It does not capture coherence, as model authors themselves acknowledge (Cohan et al. 2018).

SummEval re-scored many systems with both automatic metrics and human judgments to measure these gaps (Fabbri et al. 2021).

Evaluation: the Pyramid method and factual consistency

The Pyramid method evaluates content selection directly (Nenkova & Passonneau 2004). Annotators identify summary content units, which are small units of meaning, across several reference summaries. Each unit is weighted by how many references express it. A system summary scores the total weight of the units it contains, relative to an ideal summary of the same size. TAC 2008 reported Pyramid scores alongside ROUGE. For example, a concept-based ILP scored 0.345 against 0.186 for a lead baseline (Gillick & Favre 2009).

Factual consistency asks whether every claim in the summary is supported by the source. A manual audit sampled 200 article–summary pairs from abstractive models. It found inconsistencies in 60 of them (30%) (Kryscinski et al. 2019). Maynez and colleagues analysed how abstractive models produce content that is unfaithful to the source or untrue in the world (Maynez et al. 2020).

Several families of automatic checks followed:

• Entailment-style classifiers trained to detect inconsistency, notably FactCC (Kryscinski et al. 2020).
• Question-generation and answering approaches that test whether answers drawn from the summary match the source.
• Benchmarks that break errors into types. FRANK labelled 2,250 summaries for predicate, entity, circumstance, coreference, discourse-link and unverifiable-content errors. Correlations between metrics and humans were mostly modest, and each metric family had its own blind spots (Pagnoni et al. 2021).

These checks connect summarization to information extraction and to the broader study of evaluation and benchmarks.

Datasets and benchmarks

News: CNN/DailyMail and XSum

CNN/DailyMail pairs news articles with their bullet-point highlights. Hermann et al. (2015) introduced the corpus for reading comprehension. Nallapati and colleagues then adapted it to summarization (Nallapati et al. 2016). Cheng and Lapata built large DailyMail training sets automatically from editor-written highlights (Cheng & Lapata 2016). Later papers adopted a non-anonymized version of the data (See et al. 2017).

The benchmark has well-known weaknesses. Thirteen published systems clustered between about 39.5 and 41.6 ROUGE-1. When Lead-3 was used as the reference instead, their scores rose to roughly 52–79. This shows how much the systems lean on article layout (Kryscinski et al. 2019).

XSum asks for a single-sentence answer to "what is the article about?". Narayan, Cohen and Lapata introduced this "extreme" summarization task with a topic-aware convolutional model (Narayan et al. 2018). Its references are more abstractive. That makes factuality errors more frequent and entailment metrics less reliable (Pagnoni et al. 2021).

Long documents: arXiv and PubMed

Cohan and colleagues used papers' abstracts as targets for long-document summarization (Cohan et al. 2018):

• arXiv: 215K papers averaging about 4,938 words.
• PubMed: 133K papers averaging about 3,016 words.
• For comparison, a CNN/DailyMail article averages roughly 650 words.

These sets also have long summaries, and there whole-summary extraction gains little. Consider the gap between sentence-level and summary-level extraction. MatchSum, which scores whole candidate summaries, recovered a share of 0.64 of that gap on XSum. On PubMed it recovered only 0.09 (Zhong et al. 2020).

Multi-document sets

The DUC and TAC news collections defined multi-document evaluation for a decade (Gillick & Favre 2009; Haghighi & Vanderwende 2009). Multi-News scaled the task to 56,216 cluster–summary pairs built from newser.com. Each cluster has 2–10 sources, and summaries average about 264 words (Fabbri et al. 2019). WikiSum, a setup introduced by Liu et al. (2018), takes cited web pages and search results as input. Its target is a Wikipedia lead section. It was later reused to train Hierarchical Transformers (Liu & Lapata 2019).

Dialogue and query-focused sets

SAMSum provides human-written abstractive summaries of messenger-style conversations (Gliwa et al. 2019). QMSum pairs queries with summaries over meeting transcripts from product design, academic and parliamentary settings. It contains 1,808 query–summary pairs over 232 meetings (Zhong et al. 2021). PEGASUS was also tested on other domains, among them patents, legislative bills, emails, how-to guides and Reddit posts (Zhang et al. 2019).

People and institutions

Much of the field's shape comes from a few long research programmes.

• Mirella Lapata, now at the University of Edinburgh, co-authored neural extraction (Cheng & Lapata 2016) and the XSum task (Narayan et al. 2018). She also co-authored pretrained-encoder summarization (Liu & Lapata 2019) and Hierarchical Transformers for multiple documents (Liu & Lapata 2019).
• Regina Barzilay, now at the Massachusetts Institute of Technology's Computer Science and Artificial Intelligence Laboratory (CSAIL), worked on lexical chains (Barzilay & Elhadad 1997), information fusion (Barzilay et al. 1999) and content models (Barzilay & Lee 2004).
• Kathleen McKeown, at Columbia University, worked with Barzilay on sentence fusion for Newsblaster (Barzilay & McKeown 2005). She later co-authored the large human benchmark of language models on news (Zhang et al. 2024).
• Michael Elhadad, at Ben-Gurion University of the Negev, co-authored the lexical-chain and fusion papers (Barzilay & Elhadad 1997; Barzilay et al. 1999).
• Dragomir Radev, who was at Yale University, led the MEAD platform (Radev et al. 2004) and LexRank (Erkan & Radev 2011). He also co-authored Multi-News (Fabbri et al. 2019), QMSum (Zhong et al. 2021) and SummEval (Fabbri et al. 2021).
• Alexander Fabbri, now at Salesforce, co-authored Multi-News and SummEval with Radev (Fabbri et al. 2019; Fabbri et al. 2021).
• Caiming Xiong, at Salesforce, co-authored the reinforced abstractive model (Paulus et al. 2017) and the critical evaluation of neural summarization (Kryscinski et al. 2019). He also worked on factual-consistency checking (Kryscinski et al. 2020).
• Xipeng Qiu, at Fudan University, and Pengfei Liu co-authored MatchSum and heterogeneous graph extraction (Zhong et al. 2020; Wang et al. 2020).
• Ming Zhong, now at the University of Illinois Urbana-Champaign, led MatchSum and QMSum (Zhong et al. 2020; Zhong et al. 2021).
• Furu Wei, at Microsoft, worked with Ming Zhou on NeuSum (Zhou et al. 2018) and HIBERT (Zhang et al. 2019).
• Michael Auli, at Meta, co-authored recurrent attentive summarization (Chopra et al. 2016) and controllable summarization (Fan et al. 2018).
• Arman Cohan introduced the arXiv and PubMed long-document corpora (Cohan et al. 2018).

Chin-Yew Lin defined the field's standard metric (Lin 2004). Ani Nenkova and Rebecca Passonneau defined its main manual content protocol (Nenkova & Passonneau 2004).

Open questions

Measuring what matters

ROUGE persists despite known flaws. Proposed extensions have not replaced plain ROUGE as the usual choice (Kryscinski et al. 2019). As system outputs approach human quality, references become the bottleneck: weak references can make better systems look worse (Zhang et al. 2024).

Factuality metrics disagree with each other and fail on different error types (Pagnoni et al. 2021). A trustworthy, cheap and domain-robust measure of faithfulness remains open.

Faithfulness by design

Copying and content selection reduce hallucination but make output more extractive (Gehrmann et al. 2018). Highly abstractive targets invite unsupported content (Maynez et al. 2020). Clinical deployments show what is at stake. In the medical evaluation, both machine and expert errors were linked to potential harm (Veen et al. 2023).

Long and many inputs

Book-length documents, large clusters and long meetings still exceed what models can attend to well. Strong baselines performed poorly on query-based meeting summaries and produced off-topic or wrong content (Zhong et al. 2021). Hierarchical encoding (Liu & Lapata 2019) offers a partial answer. So do retrieval-first pipelines, such as the locate-then-summarize approach tested on QMSum (Zhong et al. 2021). That approach is close to retrieval-augmented generation. Whether long-context language models remove the need for such pipelines is unsettled.

Benchmarks, bias and the user

News benchmarks reward exploiting the position of information in the article (Kryscinski et al. 2019). Single-document news summarization may be close to saturated (Zhang et al. 2024).

Summaries are also personal. In an informal test, three people summarized the same ten articles about the Microsoft trial. They produced very different summaries, focusing in turn on courtroom detail, the gist of the day, and the trial's aims and result (Goldstein et al. 2000). Controllable and query-focused methods (Fan et al. 2018) point toward systems that ask what a particular reader needs, rather than producing one "correct" summary.

The library in figures