🗨️ Natural language processing

About 38 minutes to read · every paper named here has its own page in this library

What it is

Natural language processing (NLP) covers the study and engineering of computer programs that work with ordinary human language: the English of an email, the Arabic of a news broadcast, the Finnish of a court ruling, the transcript of a phone call. Most people meet it every day without noticing. It filters spam, suggests the next word on a phone keyboard, translates a web page, answers a spoken question and powers the conversational assistants that write essays and code on request.

The field has two neighbouring identities. Computational linguistics is the scientific side: it uses computation to model how language works and how people process it. NLP is usually the engineering side: it builds systems that perform useful tasks, whether or not they reflect how humans do the same thing. In practice the communities overlap almost completely. They share conferences, datasets and people. The work is commonly split into natural language understanding, mapping text to some representation of its meaning, and natural language generation, producing fluent text from data, meaning or other text. The concrete products built from both are catalogued under NLP applications. The families of systems that implement them are described under natural language processing models.

Language is hard for machines for reasons that are easy to state. Words are ambiguous: "bank" may be a river edge or a financial firm, and "saw" a tool or a past tense. Sentences have hidden structure, so "old men and women" can group in two ways. Meaning depends on context, on what the speaker intends and on knowledge of the world that is rarely written down. Language data is also extremely uneven. A handful of words make up much of any text, while most words are rare, the pattern described by Zipf's law. Kilgarriff and Grefenstette argued that this skew leaves even a hundred-million-word corpus too sparse for rare words, rare senses and rare word combinations (Kilgarriff & Grefenstette 2003). Much of the history of NLP is a history of ways to cope with that sparsity: hand-written rules, statistics over large corpora, dense vector representations, and finally models pretrained on enormous quantities of raw text.

Today's dominant systems are large language models: neural networks trained to predict text, which can then be steered towards almost any language task. Their success has reopened old questions about what it means for a machine to "understand". Bender and Koller argue that a system trained only on linguistic form cannot, in principle, learn meaning in the sense of the link between words and communicative intent (Bender & Koller 2020). The rest of this article explains how such systems work, where they came from, which levels of linguistic structure they deal with, what tasks they solve, how they are measured and what remains unresolved.

How it works

A modern NLP system can be described as a chain of four transformations: raw text becomes a sequence of tokens, tokens become numerical representations, a model computes over those representations, and a final component turns the result into an output such as a label, a span, a structure or new text. Older systems ran a pipeline of separate tools. The Stanford CoreNLP toolkit, for example, chains analysers that add layer after layer to a shared annotation object, from tokenization and sentence splitting through tagging, entity recognition and parsing up to coreference and sentiment (Manning et al. 2014). Current systems usually fold most of these stages into one trained network. The four conceptual steps remain visible inside it.

From text to tokens

Computers receive text as a string of characters, and the first decision is how to cut it into units. Splitting on spaces fails for Chinese and Japanese, which do not mark word boundaries. It also produces an unbounded vocabulary in languages with rich word formation. As Sennrich, Haddow and Birch pointed out, neural translation systems of the mid-2010s typically worked with a closed vocabulary of some 30–50 thousand words and stumbled on compounds, names and loanwords (Sennrich et al. 2016). They addressed this by adapting byte pair encoding (BPE). Frequent symbol pairs are merged repeatedly until a fixed-size inventory of word pieces remains, and any word can be spelled from them (Sennrich et al. 2016). This family of methods is called subword tokenization. SentencePiece made it language-independent. It learns a BPE or unigram vocabulary directly from raw sentences and treats whitespace as an ordinary symbol, so tokenization can be reversed exactly (Kudo & Richardson 2018). Kudo also showed that the ambiguity of segmentation can itself be useful. Sampling a different segmentation of each sentence at every training step works as data augmentation without changing the network (Kudo 2018). The wider literature on cutting text into words, pieces and sentences appears under tokenization and segmentation.

From tokens to representations

Classical systems treated each vocabulary item as an isolated index. As Mikolov and colleagues observed, this ignores any similarity between words (Mikolov et al. 2013). The alternative is to map each token to a dense vector, a word embedding, so that words used alike end up close together. Bengio and colleagues proposed learning such vectors jointly with a probabilistic model of the next word. In their model a single training sentence raises the probability of many similar sentences (Bengio et al. 2003). Word2vec made this cheap with two shallow log-linear models that have no non-linear hidden layer: continuous bag-of-words (CBOW) and Skip-gram (Mikolov et al. 2013). Negative sampling and subsampling of frequent words improved it further (Mikolov et al. 2013). GloVe derived vectors from ratios of global co-occurrence probabilities (Pennington et al. 2014).

These static embeddings give each word one vector regardless of context. Contextual models compute a fresh vector for every occurrence, so "bank" in a sentence about rivers differs from "bank" in a sentence about loans. ELMo, from «Deep Contextualized Word Representations», did this by joining two unidirectional language models trained separately (Devlin et al. 2019). BERT made every layer condition on both left and right context at once (Devlin et al. 2019).

The model: a language model

At the heart of most current systems is a language model, a probability distribution over sequences of tokens. By the chain rule of probability, the probability of a whole sequence factors into a product of next-token predictions:

P(w₁, w₂, …, w_T) = ∏ₜ₌₁ᵀ P(wₜ | w₁, …, wₜ₋₁)

L(θ) = − Σₜ₌₁ᵀ log P_θ(wₜ | w₁, …, wₜ₋₁)

Here w₁ … w_T are the tokens of a text of length T, and P(wₜ | w₁, …, wₜ₋₁) is the probability the model gives to token wₜ after seeing all the tokens before it. ∏ is the product over positions t from 1 to T.

The second line is the training objective. L(θ) is the cross-entropy loss, the summed negative log-probability of the actual tokens under a model with parameters θ, and training adjusts θ to make it small. The exponential of the average per-token loss is the perplexity, the usual intrinsic measure of a language model.

N-gram models approximate the conditional probability by looking only at the last few tokens. Early feedforward neural models also used a fixed window of preceding words. Recurrent and Transformer models can condition on the whole prefix, up to a context limit.

A model that predicts well also compresses well. Any next-symbol predictor can drive an arithmetic coder, so lower log-loss means shorter codes. Delétang and colleagues used this equivalence to show that large text-trained models compress images and audio better than PNG and FLAC, provided the size of the model itself is ignored (Deletang et al. 2023).

Inside the model: attention and the Transformer

The network that computes these probabilities is today almost always a Transformer. The design comes from the paper «Attention Is All You Need», which dropped recurrence and convolution in favour of attention alone (Vaswani et al. 2017). Its central operation is scaled dot-product attention:

Attention(Q, K, V) = softmax(Q Kᵀ / √dₖ) V

Q, K and V are matrices of queries, keys and values. Each row is a vector computed from one token's representation by a learned linear map. Q Kᵀ gives, for every pair of positions, a score of how relevant one token is to another. Dividing by √dₖ, the square root of the key dimension, keeps these scores in a range where the softmax does not saturate. The softmax turns each row of scores into weights that sum to one. Multiplying by V mixes the value vectors according to those weights.

Several such attention "heads" run in parallel, and positional encodings tell the model the order of tokens (Vaswani et al. 2017). Encoder-only Transformers such as BERT read a whole input at once (Devlin et al. 2019). Decoder-only models such as GPT-3 generate left to right (Brown et al. 2020). Encoder–decoder models such as T5 map input text to output text (Raffel et al. 2019).

From model to output

The last stage depends on the task:

• A classifier puts a small layer on top of a summary vector.
• A tagger labels every token.
• An extractive question-answering system scores start and end positions of a span.
• A generator samples tokens one at a time from the model's distribution.

The Hugging Face Transformers library formalises this split behind one programming interface. Each model has three parts: a tokenizer, a Transformer body that yields contextual embeddings and a task-specific head (Wolf et al. 2020).

For generation, the choice of decoding rule matters a great deal. Greedy decoding and beam search pick high-probability continuations, as in early neural dialogue models (Vinyals & Le 2015). Holtzman and colleagues showed that maximising likelihood leads to bland and repetitive text, and that sampling from a truncated distribution behaves better (Holtzman et al. 2019).

History

The history of NLP is usually told in four overlapping eras, each defined by where the knowledge in a system comes from. The boundaries are blurred. Rule-based components survived well into the statistical era, and statistical evaluation habits carried straight into the neural one.

Rules and symbolic systems

The idea that language could be handled by machine is as old as the computer. Turing (1950) proposed conversation as a test of machine intelligence. Weaver (1949) suggested that translation might be treated as decoding. Chomsky (1957) gave the field formal grammars, among them the context-free grammar, which rewrites symbols such as "sentence" into phrases and words.

Early systems were built by hand. Weizenbaum's ELIZA (1966) simulated a therapist with pattern-matching rules. Winograd's SHRDLU (1972) understood commands about a small world of blocks. Such systems could be impressive within their domain and brittle outside it. Writing rules for every construction, exception and domain did not scale.

The symbolic tradition also produced lasting tools for knowledge representation, the encoding of facts and rules in a form a program can reason with. This line later continued in ontologies and knowledge graphs. Rule-based methods remained competitive in specific niches. Brill's transformation-based learner, described under Methods below, showed that readable rules could themselves be learned from annotated data (Brill 1995).

Statistical NLP

Around 1990 the field turned to statistics. The IBM group of Brown, Jelinek and colleagues treated translation as a problem of estimating probabilities from parallel text (Brown et al. 1990). Speech recognition, where probabilistic language models were already standard, supplied many of the methods. Jelinek and Lafferty showed how to compute the probability that a stochastic context-free grammar generates a sentence beginning with a given prefix, which a recognizer needs in order to predict the next word (Jelinek & Lafferty 1991).

Word meaning and association became statistical questions too:

• Church and Hanks used mutual information to measure word association from newswire (Church & Hanks 1990).
• Yarowsky disambiguated word senses using thesaurus categories trained on encyclopedia text (Yarowsky 1992).
• He later used a bootstrapping method that rivalled supervised learning (Yarowsky 1995).

Annotated corpora became the field's infrastructure. The Penn Treebank was annotated for part of speech and skeletal syntax during a first phase from 1989 to 1992, and it gave parsers a shared training and test bed (Marcus et al. 1993). Lexicalized parsers such as Collins's head-driven models (Collins 2003) and the unlexicalized parser of Klein and Manning (Klein & Manning 2003) were trained and compared on it.

Machine learning entered every task: support vector machines for text categorization (Joachims 1998), classifiers for sentiment (Pang et al. 2002), probabilistic topic models (Blei et al. 2003). Shared evaluations shaped the era, from the Message Understanding Conferences (Grishman & Sundheim 1996) to the automatic BLEU score for translation (Papineni et al. 2002).

Neural NLP

Neural networks had been applied to language before, but the neural era is sometimes dated from Bengio's neural probabilistic language model (Bengio et al. 2003). Collobert and colleagues then showed that one network could match engineered systems on four tagging tasks. Its word vectors were pretrained as a language model on roughly 852 million words (Collobert et al. 2011). Deep convolutional networks transformed computer vision (Krizhevsky et al. 2012). Residual connections made very deep networks trainable (He et al. 2015). Both ideas fed back into language work.

The years 2013–2016 brought a run of new methods:

• word embeddings from word2vec (Mikolov et al. 2013) and GloVe (Pennington et al. 2014);
• recurrent encoder–decoders for translation (Cho et al. 2014);
• the attention mechanism, which let the decoder look back at each source word (Bahdanau et al. 2014);
• convolutional sentence classifiers (Kim 2014) and neural dependency parsers (Chen & Manning 2014);
• named-entity taggers that pair a bidirectional long short-term memory network (BiLSTM) with a conditional random field (CRF) output layer (Lample et al. 2016).

Hand-engineered features gave way to learned representations across the board.

Pretrained Transformers

The Transformer of 2017 (Vaswani et al. 2017) made it practical to train very large models on very large corpora. BERT showed that pretraining a bidirectional encoder on unlabeled text and then fine-tuning it lifted results on a broad range of tasks (Devlin et al. 2019). A 2019 tutorial noted that within roughly two years such pretrained approaches had raised the state of the art on many benchmarks (Ruder et al. 2019). T5 recast every task as text-to-text (Raffel et al. 2019). GPT-3, with 175 billion parameters, performed tasks from instructions and examples given in its prompt, without any weight update (Brown et al. 2020).

Empirical scaling laws described how loss falls with model size, data and compute (Kaplan et al. 2020). A later analysis revised the best balance between parameters and training tokens (Hoffmann et al. 2022). Several techniques then changed how such models are used:

• Fine-tuning on human feedback made models follow instructions (Ouyang et al. 2022).
• Step-by-step prompting improved multi-step reasoning (Wei et al. 2022).
• Retrieval connected generators to external documents (Lewis et al. 2020).

Levels of language

Linguistics divides language into levels, from the structure of words to the use of whole conversations. NLP inherited this division. For decades each level had its own tasks, annotated resources and specialist tools. End-to-end neural models blur the levels, but the levels remain the vocabulary in which errors are diagnosed and capabilities are tested.

Morphology

Morphology is the study of how words are built from smaller meaningful parts: roots, prefixes, suffixes, inflections for tense or case. English morphology is modest, while Turkish, Finnish or Arabic can pack a clause's worth of information into one word. Morphological analysers split words into such parts. Generators produce the right inflected form for a given set of features.

Creutz and Lagus showed that morpheme-like units can be discovered from unannotated text alone (Creutz & Lagus 2002). The SIGMORPHON 2016 shared task turned inflection into a learning benchmark: produce the target form of a word given another form and the desired features (Cotterell et al. 2016). In Universal Dependencies, morphology forms a separate annotation layer. Text is first split into syntactic words, with clitics separated and contractions undone, and each word then receives a lemma and standardized features (Nivre et al. 2020).

Work on word structure and analysers can be browsed among the morphology research papers. Languages whose words are built from consonantal roots and vowel patterns are treated under Semitic languages.

Closely tied to morphology is part-of-speech tagging, labelling each word as noun, verb, adjective and so on in its context. A part-of-speech tagging reading list covers the task's literature. Brill learned ordered correction rules on top of a simple initial tagger (Brill 1995). Toutanova and colleagues built a cyclic dependency network in which each tag decision sees the tags on both sides (Toutanova et al. 2003).

Even within one language, different treebanks may apply the same guidelines differently. Aggarwal and Zeman proposed a symmetric divergence over tag trigrams to measure how consistent part-of-speech annotation is across treebanks (Aggarwal & Zeman 2020).

Syntax and parsing

Syntax concerns how words combine into phrases and sentences. Parsing is the automatic assignment of that structure. Two representations dominate:

• Constituency trees group words into nested phrases, the structure described by context-free grammars and their richer descendants. The formalisms and the parsers built on them are listed under syntax and grammar formalisms.
• Dependency parsing instead links each word to the word it depends on, such as a subject to its verb. Its research record is in the dependency parsing papers.

Statistical constituency parsing grew up on treebanks, corpora in which every sentence carries a hand-checked tree. Collins modelled each tree as a sequence of decisions generated outward from the head word of every phrase (Collins 2003). Klein and Manning argued that much of the gain credited to word-level statistics really came from structural refinement. Their unlexicalized grammar split categories by linguistically motivated cues (Klein & Manning 2003). Kitaev and Klein later replaced the recurrent encoder of a span-based parser with self-attention layers. They found that the model relied heavily on word position and gained from keeping position and content in separate channels (Kitaev & Klein 2018).

Dependency parsing moved through the same transitions. Chen and Manning replaced millions of sparse hand-built features in a transition-based parser with a small neural network over dense embeddings. They noted that feature computation had consumed over 95% of parsing time in the older design (Chen & Manning 2014). Dozat and Manning's graph-based parser scores every possible head–dependent arc with a biaffine function over BiLSTM states (Dozat & Manning 2016).

Multilingual comparison required shared formats. The CoNLL-X shared task converted treebanks for 13 languages into one tab-separated format and fixed labeled attachment score as the official metric (Buchholz & Marsi 2006). Even then, annotation schemes differed widely. On parallel Swedish, Danish and English sentences, no two languages shared more than 40% of their dependencies (Nivre et al. 2016).

Universal Dependencies answered this with one cross-linguistically harmonized scheme (Nivre et al. 2016). Its second version, described in an LREC 2020 report, revised the guidelines while keeping the principle that relations between content words take precedence over those involving function words (Nivre et al. 2020; Nivre et al. 2020). The project's own publications are listed among the Universal Dependencies papers.

Semantics

Semantics is the study of meaning. At the level of words it is lexical semantics: senses, synonymy, hyponymy, the structure of the lexicon. Related work, including studies that treat words as vectors, sits under lexical semantics and word embeddings.

WordNet organized English nouns, verbs, adjectives and adverbs into synsets, groups of synonyms linked by relations such as hyponymy and meronymy (Miller 1994). Budanitsky and Hirst compared WordNet-based relatedness measures on a practical task, detecting real-word spelling errors (Budanitsky & Hirst 2006). Word sense disambiguation chooses which sense a word has in context; the word sense disambiguation papers trace its methods. It drew on a semantic concordance in which content words of the Brown Corpus were tagged with WordNet senses (Miller et al. 1993).

A second tradition, distributional semantics, defines a word's meaning by the company it keeps. Turney and Pantel surveyed vector space models built from corpus frequencies. They classified them by matrix type into term–document, word–context and pair–pattern models (Turney & Pantel 2010). Word embeddings are the neural continuation of this idea. Measuring semantic similarity between words or sentences is the natural test for both.

At the level of the clause, the question is who did what to whom. Semantic role labeling identifies the participants of a predicate and their roles; its development can be followed through the semantic role labeling papers. Three resources shaped the task:

• FrameNet described English words in terms of frames, each with its own frame elements, using sentences from the British National Corpus (Baker et al. 1998).
• Gildea and Jurafsky trained the first statistical role labeler on about 50,000 of these annotated sentences (Gildea & Jurafsky 2002).
• PropBank added verb-specific roles over the Penn Treebank. In it, Arg0 roughly corresponds to a proto-agent and Arg1 to a proto-patient (Palmer et al. 2005).

Whole-sentence meaning is the target of semantic parsing, which maps text to a formal representation; see semantic parsing and meaning representation. Abstract Meaning Representation writes a sentence's meaning as a rooted, directed, labelled graph built on PropBank frames (Banarescu et al. 2013).

Not all semantic theories treat meaning as a formal structure. Construction grammar holds that grammar consists of learned pairings of form and meaning, from idioms to abstract patterns. Cognitive linguistics grounds meaning in embodied experience and conceptual metaphor.

Both views have computational work behind them:

• Steels modelled how grounded constructions could emerge among interacting agents (Steels 2004).
• With de Beule he introduced Fluid Construction Grammar for robots that converse about what they perceive (Steels & Beule 2006).
• Tayyar Madabushi and colleagues found that pretraining BERT on sentences grouped by construction made no noticeable difference. They read this as a sign that much constructional information is already captured (Madabushi et al. 2020).
• Turney and colleagues used the cognitive-linguistic view of metaphor to separate literal from figurative uses by the abstractness of the surrounding words (Turney et al. 2011).

Pragmatics and discourse

Pragmatics concerns meaning in use: what a speaker intends by saying something in a given context. Speech act theory, originating with Austin (1962) and Searle (1969), treats utterances as actions such as promising, asking or ordering. Implicature, a notion due to Grice (1975), covers what is meant beyond what is said. The implicature papers trace its formal analyses.

Several of these analyses are formal models of reasoning between speaker and hearer:

• Franke derived quantity implicatures, such as reading "some" as "not all", from iterated best responses between rational speaker and hearer in a signaling game (Franke 2011).
• Frank and Goodman turned similar reasoning into a Bayesian model whose quantitative predictions matched people's bets in simple reference games (Frank & Goodman 2012).
• McCready extended a logic of conventional implicature to words such as slurs, which carry descriptive and expressive content at once (McCready 2010).
• Roberts modelled conversation as a shared inquiry organized by a stack of questions under discussion, and used it to account for prosodic focus (Roberts 2012).

Discourse analysis studies how sentences connect into coherent texts; its computational side is represented by the computational discourse analysis papers. Two annotated resources dominate:

• The Rhetorical Structure Theory (RST) Discourse Treebank built trees of rhetorical relations over clause-sized units (Carlson et al. 2001). Soricut and Marcu parsed single sentences into such trees with probabilistic models over syntactic structure (Soricut & Marcu 2003).
• The Penn Discourse TreeBank (PDTB) 2.0 instead anchored relations to connectives such as "because", over roughly a million words of Wall Street Journal text. Where no connective was written, annotators inserted an implicit one (Prasad et al. 2008).

Coreference resolution decides which expressions refer to the same entity, as when "she" points back to "the minister". The coreference resolution papers follow its methods. Soon, Ng and Lim cast it as pairwise classification with a decision tree over twelve cheap features (Soon et al. 2001). Lee and colleagues later built the first end-to-end neural model, which considers every span as a candidate mention and needs no parser (Lee et al. 2017).

Corpora and annotation

Every level above depends on data. A corpus is a collection of texts studied as language, and corpus linguistics is the discipline of drawing linguistic conclusions from such collections. Kilgarriff and Grefenstette argued that, by this definition, the Web itself is a corpus, and that its quality is a practical question about fitness for a task (Kilgarriff & Grefenstette 2003).

Annotation adds analyses to raw text. The Penn Treebank reduced the Brown Corpus tagset by dropping distinctions recoverable from the word or the tree. Its annotation was produced by an automatic first pass followed by human correction (Marcus et al. 1993).

Corpus statistics also raise theoretical puzzles. Nakaishi and colleagues tested on real parse trees whether the power-law decay of dependence between distant words really signals hierarchical structure (Nakaishi et al. 2026).

Corpora also changed lexicography, the craft of making dictionaries:

• Church and Hanks proposed the association ratio so that word association norms could be estimated automatically for tens of thousands of words (Church & Hanks 1990).
• Smadja's Xtract retrieved collocations and labelled them syntactically (Smadja 1993).
• The Leipzig Corpora Collection built frequency-based monolingual dictionaries from crawled text, with 136 languages online at the time of its 2012 report (Goldhahn et al. 2012).

Tasks and applications

NLP is organized less by theory than by tasks: well-defined input–output problems with datasets and metrics. The families below cover the main ones. Many products combine several of them.

Analysis of structure

Structural analysis covers tagging, chunking, parsing and the recognition of names, which earlier sections introduced as levels of language. Named entity recognition finds names in text and classifies them as persons, places, organizations and other types. Its literature is listed in the named entity recognition papers.

Ratinov and Roth examined the task's design choices (Ratinov & Roth 2009). They made two findings:

• The BILOU encoding of chunks, which tags beginning, inside, last and outside tokens and unit-length names, beat the common BIO scheme of beginning, inside and outside tags.
• Greedy decoding was nearly as accurate as Viterbi decoding at far lower cost.

Neural taggers then removed the need for gazetteers and hand-built features. Lample and colleagues combined a BiLSTM with a CRF layer (Lample et al. 2016). Chiu and Nichols added character-level convolutions and lexicon features built from DBpedia (Chiu & Nichols 2015). The single network of Collobert and colleagues, mentioned in the history above, had already counted entity recognition among its four tasks (Collobert et al. 2011). Coreference resolution, which links mentions across sentences, sits at the boundary between structure and discourse (Pradhan et al. 2012).

Meaning

Tasks of meaning test whether a system grasps what text says. Semantic textual similarity asks how close in meaning two sentences are. The SemEval-2017 edition rated pairs on a 0–5 scale and scored systems by Pearson correlation with human judgements (Cer et al. 2017).

Comparing every pair with a cross-encoder is costly. Reimers and Gurevych estimated that finding the most similar pair among 10,000 sentences takes about 50 million BERT passes, or roughly 65 hours on one NVIDIA V100 GPU. Their siamese Sentence-BERT produces reusable sentence vectors instead (Reimers & Gurevych 2019). SimCSE improved such vectors with contrastive learning, using two dropout masks on the same sentence as a positive pair (Gao et al. 2021). Babelfy unified word sense disambiguation and entity linking over the multilingual BabelNet network (Moro et al. 2014).

Figurative and fixed language is a meaning problem of its own. Idioms and multiword expressions mean more than their parts. Metaphor detection has drawn on concreteness and supersense features transferred across languages (Tsvetkov et al. 2014) and on visual embeddings of words and phrases (Shutova et al. 2016).

Meaning also includes inference. Work on reasoning covers entailment, commonsense and mathematics. Chain-of-thought prompting, in which demonstrations spell out intermediate steps, brought large gains on arithmetic and symbolic problems, though only in sufficiently large models (Wei et al. 2022).

Semantic parsing turns questions into executable queries:

• Berant and colleagues trained a parser for Freebase from question–answer pairs alone, treating logical forms as latent (Berant et al. 2013).
• Kwiatkowski and colleagues bridged the gap between linguistic structure and a knowledge base's schema with on-the-fly ontology matching (Kwiatkowski et al. 2013).
• Spider, written by 11 students over about 1,000 hours, tests text-to-SQL across many multi-table databases (Yu et al. 2018). It connects NLP to code and programming languages.

The targets of such parsers are structured stores of facts. Ontology engineering designs the classes and relations of a domain. A knowledge graph stores facts as a graph whose nodes are entities and whose edges are relations (Hogan et al. 2020); further work appears under knowledge graphs and ontologies. DBpedia extracts Resource Description Framework (RDF) triples from Wikipedia infoboxes and other structured page content (Lehmann et al. 2015). Wikidata is edited collaboratively (Vrandečić & Krötzsch 2014). Färber and colleagues compared the quality of five such graphs along 34 criteria (Färber et al. 2018).

Embedding methods predict missing links. TransE models relations as translations between entity vectors (Bordes et al. 2013). A simple bilinear model performed best in a unifying comparison (Yang et al. 2014).

Information access: extraction, retrieval, question answering

Information extraction pulls structured facts out of running text: relations, events and the fields of a record. A dedicated list of information extraction papers traces the task. It is the core of text mining, the large-scale discovery of patterns in document collections.

The task's methods developed in several steps:

• The Message Understanding Conferences defined early extraction tasks and their evaluation (Grishman & Sundheim 1996).
• Hearst showed that lexical patterns such as "X such as Y" reveal hyponyms in large corpora (Hearst 1992).
• Distant supervision avoided manual labels by aligning a knowledge base with text that mentions the same entity pairs (Mintz et al. 2009).
• Open information extraction went further. Its TextRunner system collected triples for arbitrary relations in a single pass over web text, with no relation named in advance (Banko et al. 2007).

Topic modeling summarizes what a collection is about without supervision. Probabilistic latent semantic analysis explained word–document co-occurrences through hidden aspects (Hofmann 1999). Latent Dirichlet allocation placed this in a full Bayesian model of documents (Blei et al. 2003). Dynamic topic models let topics drift across time slices (Blei & Lafferty 2006). The same question, how content and words shift over time, is central to historical linguistics and language change.

Information retrieval and search finds the documents relevant to a query. Question answering goes one step further and returns the answer itself; its benchmarks and systems fill the question answering papers. SQuAD built over 100,000 questions on 536 Wikipedia articles, each answered by a span of the passage (Rajpurkar et al. 2016). SQuAD 2.0 added crowd-written unanswerable questions so that systems must learn to abstain (Rajpurkar et al. 2018).

For open-domain questions, Dense Passage Retrieval replaced keyword retrieval with two BERT encoders scored by dot product and trained with in-batch negatives (Karpukhin et al. 2020). Retrieval-augmented generation couples such a retriever to a generator, so that answers draw on fetched documents rather than only on the model's parameters (Lewis et al. 2020). The LAMA probe asked the complementary question: how much factual knowledge can be read out of a pretrained model by masked cloze queries alone (Petroni et al. 2019).

Generation: translation, summarization, dialogue

Text generation covers all programs that write. Classical generators split the job into content selection, microplanning and surface realisation. SimpleNLG handles only the last, mechanical step of building, inflecting and ordering English phrases (Gatt & Reiter 2009). The WebNLG challenge treated microplanning as a whole: systems had to verbalize sets of RDF triples drawn from DBpedia (Gardent et al. 2017).

Machine translation is the oldest generation task and the one that drove most architectural change:

• Statistical translation estimated alignment and translation probabilities from parallel text (Brown et al. 1990).
• Neural encoder–decoders first compressed a sentence into one vector (Cho et al. 2014).
• Attention then let the decoder choose which source words to consult (Bahdanau et al. 2014).
• The Transformer was first evaluated on English–German and English–French data from the 2014 Workshop on Statistical Machine Translation (WMT 2014) (Vaswani et al. 2017). The deep-dive article on «Attention Is All You Need» describes it in detail.

Translation quality is most often reported with the BLEU score, which compares clipped n-gram precision against human reference translations (Papineni et al. 2002).

Text summarization shortens a document or a set of documents to what matters. More work is listed under summarization.

• Extractive methods select sentences. TextRank ranks them, or keywords, by graph centrality (Mihalcea & Tarau 2004).
• Abstractive methods write new text. Rush and colleagues applied neural attention to sentence summarization (Rush et al. 2015). See, Liu and Manning's pointer-generator network mixed copying from the source with generating from the vocabulary (See et al. 2017).

ROUGE, the standard summarization metric, counts recall-oriented overlaps of n-grams, longest common subsequences and skip-bigrams with reference summaries (Lin 2004).

Dialogue systems talk with users, either to complete tasks or for open conversation. A chatbot is the open-ended kind. Vinyals and Le trained a generic sequence-to-sequence LSTM on helpdesk logs and movie subtitles. It held surprisingly coherent conversations, though not consistent ones (Vinyals & Le 2015). Serban and colleagues added a hierarchical encoder that tracks the conversation across turns (Serban et al. 2015). Modern assistants are large language models tuned with human feedback, a recipe popularised by InstructGPT (Ouyang et al. 2022).

Text classification and sentiment analysis

Text classification assigns labels to whole texts: topic, spam or not, language, genre, reading level. Joachims argued that support vector machines suit text because its feature spaces are huge and sparse yet nearly every feature carries some signal. He tested this on Reuters-21578 and Ohsumed (Joachims 1998). Kim showed that a one-layer convolutional network over word2vec embeddings did well across seven sentence classification benchmarks (Kim 2014).

Sentiment analysis, the detection of opinions and feelings, is the best-known classification task; see sentiment and emotion.

• Pang, Lee and Vaithyanathan found that standard classifiers on movie reviews beat human-chosen word lists, yet sentiment proved harder than topic classification (Pang et al. 2002).
• Socher and colleagues built a sentiment treebank labelled at every phrase and modelled composition with recursive networks (Socher et al. 2013).
• WordNet-Affect attached emotion labels to word senses rather than word forms (Strapparava & Valitutti 2004).

Other classification problems serve education and professional domains. Vajjala and Meurers assigned reading levels using measures from second-language acquisition research, on a corpus spanning ages 7 to 16 (Vajjala & Meurers 2012). Detecting and correcting errors in learners' writing is the subject of grammatical error correction and learner language. Classifying and analysing court decisions, statutes and contracts forms the field of legal AI. Text increasingly comes with images. Captioning, visual question answering and joint models of pictures and words make up vision and language.

Speech and psycholinguistics

Speech and spoken language processing recognizes and synthesizes speech. It also shares its oldest tool, the language model, with text NLP. A recognizer must predict the next word from those already heard. This is why Jelinek and Lafferty derived prefix probabilities for stochastic context-free grammars (Jelinek & Lafferty 1991).

The same quantity links machines to human processing. Psycholinguistics studies how people read, hear and learn language; its computational strand is covered under psycholinguistics and language acquisition.

• Hale proposed that the processing difficulty of a word is proportional to its surprisal under a probabilistic grammar, computed with an Earley parser (Hale 2001).
• Goodkind and Bicknell found that the power of surprisal to predict gaze durations grows linearly with the quality of the language model that estimates it (Goodkind & Bicknell 2018).
• Ettinger adapted psycholinguistic tests to show where BERT's predictions diverge from human expectations, for example under negation (Ettinger 2020).

Acquisition research asks what a learner can extract from realistic input. Child-directed speech is the simplified register adults use with young children. BabyBERTa, a small RoBERTa trained on transcripts of such speech, learned a good deal of English grammar from child-scale data (Huebner et al. 2021). Warstadt and Bowman note that standard language models see roughly a thousand times more text than a child. They propose ablation experiments to test which innate biases learning really needs (Warstadt & Bowman 2022).

Multilingual and low-resource languages

Most NLP research has been about English and a few other well-resourced languages. Multilingual and cross-lingual NLP builds models and resources that cover many languages at once and transfer between them. Shared annotation has been the main enabler:

• CoNLL-X ran parsing in 13 languages (Buchholz & Marsi 2006).
• The CoNLL-2012 task covered coreference in English, Chinese and Arabic (Pradhan et al. 2012).
• SemEval-2017 moved semantic similarity towards Arabic, Spanish and Turkish pairs (Cer et al. 2017).
• Universal Dependencies made treebanks of many languages directly comparable (Nivre et al. 2016).

Language-independent subword tokenizers reduced reliance on separate, language-specific word segmenters (Kudo & Richardson 2018). Tsvetkov and colleagues showed that a metaphor classifier trained on English transfers to Spanish, Farsi and Russian through concept-level features (Tsvetkov et al. 2014).

Low-resource languages are those with little digital text, including many endangered and indigenous ones. Corpus collections that crawl the web for "low density" languages are one response (Goldhahn et al. 2012). Historical and classical languages form a related strand, with small corpora, rich morphology and scholarly traditions behind them. They include Akkadian and cuneiform, Latin, Ancient Greek, and the languages grouped as Hebrew and Aramaic. The last group divides further into Biblical Hebrew, Modern Hebrew and Aramaic. For Hebrew and Arabic, segmenting a written word into its prefixes, stem and clitics is itself a hard first step.

Methods

NLP methods fall into five broad families. They are listed roughly in the order they appeared, though older families remain in use inside newer ones.

Rules

Rule-based systems encode linguistic knowledge directly: grammars, lexicons, patterns and transformation rules. Their strengths are transparency and control. A developer can see why a decision was made and change it. Their weakness is coverage, since every construction and exception has to be anticipated.

Rules survive where control matters:

• Surface realisers such as SimpleNLG leave content decisions to the developer and apply grammatical rules for inflection and ordering (Gatt & Reiter 2009).
• Brill's learner shows that rules and data need not conflict. It learns an ordered list of readable transformations by greedily reducing errors against an annotated corpus (Brill 1995).
• Hybrid systems that combined rules with machine learning won the CoNLL-2012 coreference task (Pradhan et al. 2012).

Statistical learning

The statistical turn replaced rules with probabilities estimated from corpora. A representative quantity is pointwise mutual information, the basis of Church and Hanks's association ratio (Church & Hanks 1990):

PMI(x, y) = log₂ [ P(x, y) / ( P(x) · P(y) ) ]

P(x, y) is the probability that words x and y occur together, for instance with x preceding y within a short window. P(x) and P(y) are their separate probabilities. If the two words occur independently, the ratio is 1 and the PMI is 0. Strongly associated pairs score well above 0. Counting co-occurrences in this way also underlies the term–document and word–context matrices of vector space semantics (Turney & Pantel 2010).

Generative sequence models, such as hidden Markov models, estimate how tags generate words. Discriminative models, such as maximum entropy, perceptrons, support vector machines and conditional random fields, learn to separate correct from incorrect analyses using rich features. Typical examples are:

• feature-rich log-linear tagging (Toutanova et al. 2003);
• margin-based text categorization (Joachims 1998);
• averaged perceptrons for entity recognition (Ratinov & Roth 2009).

Latent-variable models trained with expectation–maximization or variational inference discovered hidden structure, as in topic models (Hofmann 1999; Blei et al. 2003). Bootstrapping stretched small amounts of supervision a long way. Yarowsky's sense disambiguator grew from a few seed collocations by exploiting two regularities: a word tends to keep a single sense throughout a document, and nearby words point consistently to that sense (Yarowsky 1995).

Neural networks

Deep learning replaced hand-designed features with learned representations computed by multi-layer networks. In NLP this meant several kinds of network:

• word embeddings (Mikolov et al. 2013);
• convolutional networks over word windows (Collobert et al. 2011);
• recurrent networks, especially long short-term memory networks (LSTMs), that read a sequence token by token;
• recursive networks, which composed meaning along parse trees (Socher et al. 2013);
• encoder–decoder networks, which mapped one sequence to another (Cho et al. 2014), with attention to align them (Bahdanau et al. 2014).

Self-attention eventually displaced recurrence (Vaswani et al. 2017). Neural methods also unified structured prediction. Transition-based parsers (Chen & Manning 2014), graph-based parsers (Dozat & Manning 2016) and span-based coreference (Lee et al. 2017) all share the same encoders.

Pretraining and fine-tuning

Transfer learning trains a model once on a general objective and then adapts it to many tasks (Ruder et al. 2019). Pretraining needs no labels. A language model learns by predicting the next token, and BERT learns by reconstructing masked tokens (Devlin et al. 2019). Fine-tuning then trains the whole network briefly on labelled data for the target task. T5 mapped this design space with a systematic comparison of objectives, architectures, corpora and fine-tuning strategies. It used the Colossal Clean Crawled Corpus (C4), a cleaned collection of English web text (Raffel et al. 2019).

Few-shot learning, learning a task from a handful of examples, was popularised in deep learning by metric-based networks. These compare a query against class prototypes or labelled support examples (Snell et al. 2017; Vinyals et al. 2016).

Scale has its own regularities:

• Kaplan and colleagues found that test loss falls as a power law in parameters, data and compute, each when the other two are not limiting (Kaplan et al. 2020).
• Hoffmann and colleagues trained over 400 models, from about 70 million to over 16 billion parameters. They concluded that parameters and training tokens should grow in equal proportion, and that many large models of the time had been trained on too little data (Hoffmann et al. 2022).
• Sorscher and colleagues showed that careful data pruning can, in principle, beat power-law scaling (Sorscher et al. 2022).

Analysing what pretrained models learn is a method in its own right. Probing classifiers test whether representations encode a property. Pimentel and colleagues framed probing as estimating mutual information (Pimentel et al. 2020). Voita and Titov proposed using minimum description length, the cost of transmitting the labels given the representations, in place of probe accuracy (Voita & Titov 2020).

Prompting

With large enough models, adaptation can happen entirely in the input. GPT-3 performed tasks zero-shot, one-shot and few-shot from instructions and examples placed in its context, which its authors called in-context learning (Brown et al. 2020). Prompt engineering is the practice of designing such inputs. Chain-of-thought demonstrations, which show intermediate reasoning before the answer, are a prominent example (Wei et al. 2022).

Instruction tuning and reinforcement learning from human feedback make models responsive to plain requests. InstructGPT followed three steps (Ouyang et al. 2022):

1. Forty hired labelers wrote demonstrations and ranked model outputs.
2. A reward model was trained on the rankings.
3. The policy was optimized against that reward model.

Prompting is often combined with retrieval, so that relevant documents are placed in the context before the model answers (Lewis et al. 2020).

Datasets and benchmarks

Benchmarks are the field's shared measuring sticks. They fix a dataset, a split and a metric so that systems built by different groups can be compared. Shared tasks, in which teams build systems for a common test set under a deadline, have structured NLP since the Message Understanding Conferences (Grishman & Sundheim 1996). Research on how systems are measured, and on what the numbers mean, is grouped under evaluation and benchmarks.

Generation is hard to score because many outputs can be correct. Three overlap metrics became standard:

• BLEU's clipped n-gram precision with a brevity penalty became the default for translation (Papineni et al. 2002).
• METEOR added recall and matched stems and WordNet synonyms through an explicit alignment (Banerjee & Lavie 2005).
• ROUGE adapted the overlap idea to summarization with an emphasis on recall (Lin 2004).

These metrics exist because human evaluation is expensive. Lin cites a moderate summarization campaign that needed more than 3000 person-hours of judgement (Lin 2004).

Their validity is contested. Reiter's structured review gathered 284 correlations between BLEU and human judgements from 34 papers. He concluded that BLEU is acceptable for quick diagnostic comparison of translation systems but not as a primary measure elsewhere, notably for generation (Reiter 2018).

Other resources serve as infrastructure rather than tests. Lexical databases such as WordNet (Miller 1994), knowledge bases such as DBpedia (Lehmann et al. 2015) and Wikidata (Vrandečić & Krötzsch 2014), and pretraining corpora such as C4 (Raffel et al. 2019) supply the knowledge and text that models learn from.

Probing suites are a newer kind of benchmark. Rather than measuring how well a model performs an end task, they test whether a pretrained model holds facts (Petroni et al. 2019) or behaves like human readers on targeted contrasts (Ettinger 2020).

People and institutions

NLP grew from several research cultures: speech recognition laboratories, linguistics departments, artificial intelligence groups and, more recently, industrial research labs. A few individuals stand at the origin of methods that every practitioner uses:

• Fred Jelinek's group brought probabilistic modelling from speech to language (Jelinek & Lafferty 1991; Brown et al. 1990).
• Mitchell Marcus led the construction of the Penn Treebank (Marcus et al. 1993).
• George A. Miller led WordNet at Princeton (Miller 1994).
• Charles J. Fillmore's frame semantics became FrameNet (Baker et al. 1998).
• Martha Palmer led PropBank (Palmer et al. 2005).
• David Yarowsky's statistical sense disambiguation shaped the 1990s (Yarowsky 1992; Yarowsky 1995).
• Yoshua Bengio co-authored both the neural language model (Bengio et al. 2003) and neural attention for translation (Bahdanau et al. 2014).
• Tomas Mikolov and colleagues at Google introduced word2vec (Mikolov et al. 2013).
• Ashish Vaswani and colleagues designed the Transformer (Vaswani et al. 2017).
• Jacob Devlin and colleagues created BERT (Devlin et al. 2019).

Among authors who have contributed many papers to the field, Christopher D. Manning of Stanford University co-authored work spanning several eras. It includes:

• unlexicalized parsing (Klein & Manning 2003);
• feature-rich tagging (Toutanova et al. 2003);
• GloVe (Pennington et al. 2014);
• neural dependency parsing (Chen & Manning 2014; Dozat & Manning 2016);
• the CoreNLP toolkit (Manning et al. 2014);
• Universal Dependencies (Nivre et al. 2016).

Joakim Nivre, at Uppsala University, led the Universal Dependencies reports (Nivre et al. 2016; Nivre et al. 2020). His co-authors on the first version included Yoav Goldberg, of Bar-Ilan University and the Allen Institute for Artificial Intelligence, and Reut Tsarfaty of Bar-Ilan University (Nivre et al. 2016).

Other prolific authors include:

• Luke Zettlemoyer, of the University of Washington and Meta, who co-authored end-to-end neural coreference (Lee et al. 2017) and scalable semantic parsing (Kwiatkowski et al. 2013);
• Yejin Choi, who co-authored the study of neural text degeneration (Holtzman et al. 2019);
• Dan Roth, at the University of Pennsylvania, who co-authored the analysis of named entity recognition design (Ratinov & Roth 2009) and BabyBERTa (Huebner et al. 2021);
• Nathan Schneider of Georgetown University and Philipp Koehn of Johns Hopkins University, who were among the authors of Abstract Meaning Representation (Banarescu et al. 2013);
• Nizar Habash, at New York University and its Abu Dhabi campus, who works on Arabic language processing.

The list of prolific contributors continues with:

• Noah A. Smith of the Allen Institute;
• Graham Neubig;
• Massimo Poesio of Queen Mary University of London;
• Vincent Ng;
• Michael Strube of the Heidelberg Institute for Theoretical Studies;
• Furu Wei of Microsoft;
• Yue Zhang of Westlake University;
• Yang Liu of Tsinghua University.

Academic centres with large NLP groups include Carnegie Mellon University, Stanford, the University of Washington, the University of Edinburgh, the Massachusetts Institute of Technology, Peking University, Zhejiang University, the Chinese Academy of Sciences and the Mohamed bin Zayed University of Artificial Intelligence.

Industrial laboratories have produced many of the landmark models:

• Google produced word2vec (Mikolov et al. 2013) and T5 (Raffel et al. 2019).
• DeepMind, since merged into Google DeepMind, carried out the compute-optimal scaling study (Hoffmann et al. 2022).
• OpenAI produced GPT-3 (Brown et al. 2020) and InstructGPT (Ouyang et al. 2022).
• IBM produced BLEU (Papineni et al. 2002).
• Microsoft Research co-authored knowledge-base embeddings (Yang et al. 2014).
• Amazon and Meta also maintain large research groups.

Hugging Face's open library has become the common software layer for pretrained models (Wolf et al. 2020).

Open questions

Evaluation

Benchmarks saturate quickly, and high scores do not always mean what they seem to. Automatic generation metrics correlate unevenly with human judgement (Reiter 2018). They reward overlap with a reference rather than correctness or usefulness. Reading comprehension datasets taught models to always produce an answer until unanswerable questions were added (Rajpurkar et al. 2018), which shows how dataset design shapes what is learned.

Large language models raise further problems. Test sets may leak into pretraining data. Prompts change results. Open-ended outputs need human or model-based judges whose own reliability is uncertain.

How to measure robustness, factual accuracy and the absence of harmful behaviour is an active area. It intersects with bias, fairness and safety: InstructGPT was motivated in part by models that invent facts and produce toxic text (Ouyang et al. 2022). Evaluation also includes asking what models represent internally. Interpretability research debates whether probing measures knowledge in a representation or the probe's ability to learn the task (Pimentel et al. 2020; Voita & Titov 2020).

Multilinguality

Most of the world's languages have little digital text, few annotated resources and almost no benchmarks. Universal Dependencies and shared tasks have widened coverage (Nivre et al. 2020). Still, annotation consistency varies even within one language (Aggarwal & Zeman 2020). Web crawls favour well-resourced languages (Goldhahn et al. 2012).

Several questions remain open:

• How much knowledge transfers from high-resource to low-resource languages?
• Do subword vocabularies learned mostly from English treat morphologically rich languages fairly (Sennrich et al. 2016; Kudo & Richardson 2018)?
• How should tools serve dialects, historical languages and communities whose priorities differ from those of the researchers building them?

Meaning versus form

Whether language models understand language remains contested. Bender and Koller argue that learning from form alone cannot yield meaning, since meaning relates expressions to intents and to the world outside text (Bender & Koller 2020). Others point to the knowledge that can be read out of models (Petroni et al. 2019) and to their reasoning when prompted to work step by step (Wei et al. 2022). Psycholinguistic diagnostics show both human-like sensitivities and clear failures, for example with negation (Ettinger 2020).

Acquisition studies add a further puzzle. Children learn from orders of magnitude less input than models do (Warstadt & Bowman 2022), yet small models trained on child-scale speech still learn much of grammar (Huebner et al. 2021). How grounding in perception, action and social interaction would change this picture is open. Early proposals came from construction-grammar work with robots (Steels & Beule 2006) and from visual features for figurative language (Shutova et al. 2016).

Efficiency

Large models are expensive to train and to run. Scaling studies show steady gains with compute (Kaplan et al. 2020), but they also show that compute can be wasted on models too large for their data (Hoffmann et al. 2022). Better data selection may bend the curve (Sorscher et al. 2022). Quantization, pruning, distillation and sparse computation, the subject of efficient models and compression, aim to deliver similar quality at a fraction of the cost. The hardware side, from GPUs to parallel training algorithms, is covered under GPU and parallel computing.

Efficiency at inference time also has an algorithmic side. Precomputed sentence embeddings turn a quadratic search over sentence pairs into a cheap comparison of vectors (Reimers & Gurevych 2019). Retrieval can supply knowledge that would otherwise have to be stored in parameters (Lewis et al. 2020; Karpukhin et al. 2020).

Two concrete questions follow. It is not yet known whether smaller, retrieval-backed or more data-efficient systems can match the largest models on the same tasks. It is also unclear how their energy use and their accessibility to researchers outside large laboratories compare.

The library in figures