🤖 Artificial Intelligence

A survey · about 17 minutes to read · every paper named here has its own page in this library

Natural language processing (NLP) is the branch of computer science and artificial intelligence that builds programs able to analyse, understand, generate and translate human language. It sits where linguistics, statistics and machine learning meet, and for most of its history it has also been called computational linguistics: the first name stresses the engineering of working systems, the second the use of computation to study language itself. In practice the two communities share one set of conferences, one journal tradition and, increasingly, one toolbox.

In this library the Artificial Intelligence shelf, a subfield of computer science in OpenAlex, is above all natural language processing: it gathers the core literature of the field, mostly from the ACL Anthology and arXiv. It runs from early work on discourse structure and rule-based tagging, through the statistical turn of the 1990s and 2000s, to word embeddings, neural sequence models, the Transformer and the large pretrained language models that now dominate research. Along the way it covers the classic tasks: parsing, tagging, information extraction, sentiment analysis, question answering, dialogue, summarization, and the benchmarks and evaluation methods that made progress measurable.

The article below surveys the field as a whole. For depth on a single area, follow the topic pages: Machine translation, Language models & pretraining, Evaluation & benchmarks, Sentiment & emotion, Question answering and the others linked throughout.

What it is

NLP treats text and speech as data to be processed by algorithms. Its central difficulty is that language is ambiguous at every level. A word can carry several senses, a sentence can admit several syntactic analyses, a pronoun can point to more than one antecedent, and the intended meaning of an utterance depends on context, world knowledge and the speaker's goals. Much of the field's history is a search for representations and learning methods that resolve this ambiguity reliably and at scale.

The field is traditionally divided by linguistic level:

• Morphology and tokenization — splitting text into words and subword units and analysing their internal structure; see Morphology and Tokenization & segmentation.
• Syntax — assigning part-of-speech tags and building phrase-structure or dependency trees; see Part-of-speech tagging, Dependency parsing and Syntax & grammar formalisms.
• Semantics — word meaning, semantic roles and formal meaning representations; see Lexical semantics & word embeddings, Semantic role labeling and Semantic parsing & meaning representation.
• Discourse and pragmatics — how sentences connect into coherent texts and conversations; see Discourse analysis and Coreference resolution.

Alongside the levels stand applications that cut across them: translation, extraction of facts from text, classification of opinions, answering questions, holding conversations and writing summaries.

NLP differs from its neighbours mainly in emphasis. Theoretical linguistics asks how language works; NLP asks how to make a machine handle it, and will accept a method that works even if it explains little. General Artificial Intelligence supplies the learning algorithms, but NLP contributes its own problems (structured prediction over trees and sequences, open vocabularies, discrete symbols) and its own evaluation culture. Speech processing deals with the acoustic signal and is usually treated as a sister field, though the line has blurred since end-to-end neural models.

History and milestones

Symbolic beginnings

The earliest NLP systems, from the 1950s to the 1980s, were built by hand: grammars, lexicons and rules written by linguists and programmers. This era produced influential theories of how texts hang together. Barbara Grosz and Candace Sidner's Attention, Intentions, and the Structure of Discourse (1986) split discourse structure into three interacting components — the linguistic segments, the speaker's purposes, and a focus of attention — and the later Centering, a framework for local coherence described how the entity in focus shifts from one utterance to the next. Finite-state methods, surveyed in Finite-State Transducers in Language and Speech Processing, gave an efficient formal basis for morphology and speech.

The statistical revolution

From the late 1980s, the availability of large machine-readable corpora and annotated resources such as the Penn Treebank (released in the early 1990s) moved the field towards learning from data. Kenneth Church's stochastic parts program (1988) showed that a simple probabilistic model could tag unrestricted text with high accuracy. Eric Brill answered with a rule-based tagger whose rules were themselves learned from errors, later generalised as transformation-based error-driven learning. Gale and Church's sentence aligner for bilingual corpora and the IBM translation models opened the way to statistical machine translation.

The 1990s also brought a general-purpose learning framework: A maximum entropy approach to NLP (1996) argued for log-linear models that combine many overlapping features, an approach soon applied to tagging in Ratnaparkhi's maximum entropy tagger. Language modelling matured with careful comparisons of smoothing techniques. Lexicalised probabilistic parsers, notably Michael Collins's Three Generative, Lexicalised Models for Statistical Parsing and its journal version Head-Driven Statistical Models for Natural Language Parsing, set the standard for English parsing for a decade.

Feature engineering and structured learning (2000s)

The 2000s were the age of discriminative models with rich, hand-designed features: maximum entropy taggers, conditional random fields, support vector machines and kernel methods. Accurate parsers became widely usable — Accurate Unlexicalized Parsing and the split-merge grammars of Learning Accurate, Compact, and Interpretable Tree Annotation are representative. Statistical machine translation moved from word alignment (Improved Statistical Alignment Models) to discriminative log-linear systems (Och and Ney, 2002) and to hierarchical phrase-based translation. Sentiment analysis was born as a research area in 2002, and shared tasks such as CoNLL 2007 on dependency parsing made multilingual comparison routine.

Neural networks and embeddings (2010s)

Around 2011–2013 neural methods arrived in force. Dense word vectors, popularised by work such as Linguistic Regularities in Continuous Space Word Representations, replaced sparse one-hot features. Recursive and recurrent networks were applied to sentiment (Socher et al., 2011), to compositional semantics (Socher et al., 2012) and to translation (Recurrent Continuous Translation Models). Attention mechanisms, introduced for neural machine translation in 2014–2015, spread to classification (Hierarchical Attention Networks for Document Classification), relation extraction and reading comprehension. Large crowdsourced datasets — SQuAD above all — gave these data-hungry models something to learn from.

Transformers and pretrained language models (2017 onwards)

The Transformer architecture (2017) replaced recurrence with self-attention and proved both faster to train and more accurate; refinements such as relative position representations followed quickly. From 2018, pretraining a large network on raw text and then fine-tuning it became the default recipe: ELMo, BERT, GPT and their successors, including BART, ERNIE 2.0 and GLM. Shared software such as the Hugging Face Transformers library and fairseq made these models available to everyone. With GPT-3 (2020) attention shifted to prompting and in-context learning, and since 2022 instruction-tuned conversational models have reshaped both the research agenda and public perception of the field. Surveys such as A Comprehensive Overview of Large Language Models chart this latest phase.

Main problems and tasks

NLP research is organised largely around tasks: well-defined input–output problems with shared data and metrics. The most important families are these.

Sequence labelling. Assigning a label to every token: part-of-speech tags, named-entity labels, chunk boundaries. Classic systems include the cyclic dependency network tagger and the CRF-based entity recogniser of Finkel, Grenager and Manning; neural successors include contextual string embeddings and lattice LSTMs for Chinese. See Named entity recognition.

Parsing. Recovering syntactic structure, either as constituency trees or as head–dependent relations. Parsing is the subject of Dependency parsing, Treebanks and Universal Dependencies.

Semantic analysis. Labelling who did what to whom (Automatic Labeling of Semantic Roles), disambiguating word senses (Word sense disambiguation), measuring word similarity, and mapping sentences to logical forms.

Information extraction. Turning free text into structured records: entities, relations, events. Work here moved from kernels over dependency paths (Bunescu and Mooney) to convolutional and recurrent networks and graph convolutions over trees. See Information extraction and Knowledge graphs & ontologies.

Text classification and sentiment. Assigning labels to whole documents or sentences — topic, polarity, emotion, stance, toxicity. Sentiment analysis is one of the largest strands on this shelf; see Sentiment & emotion.

Question answering and reading comprehension. Finding or generating answers from a passage, a document collection or a model's own parameters; see Question answering and Information retrieval & search.

Generation. Producing fluent text: summarization, dialogue, data-to-text and open-ended text generation.

Translation. Mapping between languages, historically the founding problem of the field; see Machine translation and the separate Machine translation topic.

Methods and schools

Rule-based and knowledge-based approaches

Early systems encoded linguistic knowledge explicitly: grammars, morphological analysers, semantic networks and dictionaries such as WordNet. Hand-built resources remain valuable where data is scarce, and lexicon-based sentiment scoring — see Lexicon-Based Methods for Sentiment Analysis — still offers transparent, domain-robust baselines. Measures of word similarity over WordNet, compared in Budanitsky and Hirst and built on formulas such as Wu and Palmer's, illustrate how knowledge bases fed quantitative methods.

Statistical and distributional approaches

The statistical school holds that the right analysis is the most probable one given data. Its core tools were n-gram language models, hidden Markov models, probabilistic context-free grammars, log-linear models and, for scale, efficient toolkits such as KenLM. In parallel, distributional semantics — the idea that a word is characterised by the contexts it occurs in — produced distributional clustering, automatic thesaurus construction and word sense discrimination by context vectors, the direct ancestors of word embeddings.

A recurring concern of this school was annotation quality: if systems are trained and judged against human labels, those labels must be reliable. Jean Carletta's argument for the kappa statistic and the later survey on inter-coder agreement set the methodological standard.

Neural approaches

Neural NLP replaced manual feature design with learned representations. The progression ran from fixed word vectors to convolutional and recurrent encoders, attention, and finally Transformers pretrained on very large corpora. Pretrained models are now adapted by fine-tuning, by lightweight adapters, or by prompting — the last explored in AutoPrompt, cloze-style few-shot learning, better few-shot fine-tuning and P-Tuning.

The main debates

Several arguments run through the field. One is structure versus scale: whether explicit linguistic structure (trees, roles, logical forms) still helps once models are large enough. Probing studies such as BERT Rediscovers the Classical NLP Pipeline, a structural probe for syntax and tests of syntactic agreement in LSTMs found that much classical structure is learned implicitly, though unevenly. A second is form versus meaning: Climbing towards NLU argues that a system trained only on text form cannot in principle acquire meaning. A third concerns explanation: whether attention weights reveal why a model decided, debated in Attention is not Explanation and Attention is not not Explanation; see Interpretability. A fourth concerns cost and harm, from the energy use of training (Energy and policy considerations for deep NLP) to social bias (Language (Technology) is Power).

Datasets, resources and benchmarks

Shared data has driven NLP as much as algorithms. Key resources include:

• Treebanks and annotated corpora. The Penn Treebank for English syntax; PropBank, which adds predicate–argument labels on top of it; and the cross-lingual Universal Dependencies treebanks.
• Lexical resources. WordNet, the Paraphrase Database, and crowd-built emotion lexicons.
• Parallel corpora such as OpenSubtitles2016, and web-scale text more generally, discussed early in the Web-as-corpus special issue.
• Reading comprehension and QA: SQuAD and SQuAD 2.0, TriviaQA, Natural Questions, HotpotQA, CoQA, QuAC and BoolQ.
• Understanding benchmarks: GLUE bundled sentence-level tasks into one leaderboard.
• Fact checking and truthfulness: FEVER, LIAR and TruthfulQA.
• Sentiment and social media: movie-review corpora, the SemEval tasks on aspect-based sentiment and sentiment in Twitter, and offensive language.
• Tools: NLTK for teaching, brat for annotation, OpenNMT and fairseq for sequence modelling.

Evaluation metrics form their own literature: BLEU for translation, and ROUGE for summaries, whose n-gram overlap scoring was validated in Lin and Hovy (2003). Newer metrics try to escape reference texts altogether, such as CLIPScore for captions and FActScore for factual precision. See Evaluation & benchmarks.

Landmark papers in this library

Transformers library (2020)

Transformers: State-of-the-Art Natural Language Processing is not a new model but shared infrastructure. It wraps dozens of pretrained architectures behind one interface — tokenizer, encoder body, task head — and ties them to a community hub of downloadable weights. Its influence is visible in how quickly new models now spread from paper to practice.

BART (2020)

BART trains an encoder–decoder Transformer to reconstruct text that has been deliberately corrupted, for instance by shuffling sentences and replacing spans with a single mask. Because it combines a bidirectional encoder with a left-to-right decoder, the same model performs well on both classification and generation, especially abstractive summarization.

SQuAD (2016)

SQuAD asked crowdworkers to pose questions about Wikipedia paragraphs and mark the answer span. It was large enough to train neural readers and easy to score, so it became the reference benchmark for reading comprehension; the follow-up SQuAD 2.0 added questions with no answer in the text.

Thumbs up? (2002)

Thumbs up? framed the polarity of a movie review — favourable or not — as a supervised learning problem, with labels taken from star ratings. Learned classifiers outperformed word lists written by people, yet fell short of topic-classification accuracy — evidence that opinion is harder to read than subject matter. With Turney's unsupervised method the same year, it launched sentiment analysis as a field.

GLUE (2018)

GLUE collected nine sentence-understanding tasks — inference, paraphrase, similarity, acceptability, sentiment — into a single benchmark with a diagnostic set. It became the scoreboard on which BERT-era models were compared, and its rapid saturation prompted harder successors.

Maximum entropy models (1996)

A maximum entropy approach to NLP showed how to build a probability model from arbitrary features while assuming nothing beyond what the data supports. Its feature-selection procedure and applications to translation subproblems made log-linear modelling the workhorse of the 2000s.

Accurate Unlexicalized Parsing (2003)

Accurate Unlexicalized Parsing demonstrated that a plain treebank grammar, improved by systematically splitting its categories, could approach the accuracy of heavily lexicalised parsers. The result shaped the Stanford parser and led to the automatic state-splitting methods that followed.

Learning Word Vectors for Sentiment Analysis (2011)

Learning Word Vectors for Sentiment Analysis learned word representations that capture both meaning and sentiment, and introduced the large IMDB movie-review dataset that remains a standard test for document classification.

Hierarchical Attention Networks (2016)

Hierarchical Attention Networks for Document Classification mirrored the structure of documents: words build sentences, sentences build the document, and attention at each level picks out the parts that matter. It was among the most widely adopted neural text classifiers before pretraining took over.

Energy and Policy Considerations (2019)

Energy and policy considerations for deep NLP estimated the electricity, carbon emissions and cloud cost of training large NLP models, including architecture search. It made the environmental and economic cost of research a mainstream topic and called for reporting training cost alongside accuracy.

The Proposition Bank (2005)

The Proposition Bank added verb-specific semantic roles to the Penn Treebank, giving shallow semantic analysis a common training resource. Together with Gildea and Jurafsky's role labeller it defined semantic role labelling as a task.

Recognizing Contextual Polarity (2005)

Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis observed that a word's prior polarity often flips or vanishes in context — through negation, intensifiers or neutral use. Its two-step classifier and subjectivity lexicon became standard resources for fine-grained opinion analysis.

Natural Questions (2019)

Natural Questions used real anonymised search-engine queries paired with Wikipedia pages, annotated with long and short answers. Because the questions were asked by people who did not know the answer, it offered a more realistic test of open-domain QA. It became a standard evaluation for the retrieve-then-read line of systems that began with Reading Wikipedia to Answer Open-Domain Questions and continued with retrieval-augmented generative readers.

A Diversity-Promoting Objective (2016)

A diversity-promoting objective for conversation models tackled the tendency of neural chatbots to answer everything with bland, safe replies. Ranking candidate responses by mutual information with the input yielded more varied and relevant output, a theme later developed in persona-grounded dialogue and open-domain chatbot recipes.

People

• Christopher D. Manning (Stanford University) is the most prolific author on this shelf. His papers here span unlexicalised parsing, feature-rich tagging, CRF-based entity recognition with long-distance constraints and, later, multi-hop question answering with HotpotQA.
• Dan Klein (University of California, Berkeley) co-authored the unlexicalised parser and developed latent-annotation grammars with Slav Petrov, together defining the high-water mark of statistical constituency parsing.
• Daniel Jurafsky (Stanford University) is represented by the founding paper on automatic semantic role labelling, by diachronic word embeddings measuring semantic change, and by neural dialogue generation.
• Eduard Hovy (The University of Melbourne) appears through the ROUGE evaluation study, early opinion mining, pattern-based question answering and hierarchical attention for documents.
• Regina Barzilay (MIT) contributed lexical-chain summarization, paraphrase extraction, the entity-grid model of coherence with Mirella Lapata, and rationale-based neural explanation.
• Mirella Lapata (University of Edinburgh) works on summarization, coherence and semantic space models, including early neural extractive summarizers.
• Martha Palmer led the PropBank annotation effort and co-authored the Wu–Palmer similarity measure.
• Luke Zettlemoyer (University of Washington, Meta) is behind BART, TriviaQA, QuAC and a study of why demonstrations help in-context learning.
• Noah A. Smith (Allen Institute) has papers here on toxicity in generated text, racial bias in hate-speech detection, probing contextual representations and the interpretability of attention.
• Yejin Choi (Stanford University) co-authored RealToxicityPrompts, CLIPScore and studies of misinformation language.
• Percy Liang (Stanford University) created SQuAD and its unanswerable-question extension, and has more recently benchmarked large models on summarization.
• Nizar Habash (New York University Abu Dhabi) is the leading figure in Arabic NLP here: morphological tagging and tokenisation, preprocessing for translation, and shared tasks on dialect identification.
• Reut Tsarfaty (Bar-Ilan University) works on parsing morphologically rich languages, especially Hebrew, from joint morpho-syntactic models to the AlephBERT language model; Avi Shmidman (Bar-Ilan) builds Hebrew diacritisation tools and Hebrew BERT and LLM models.
• Iryna Gurevych (Technische Universität Darmstadt) co-authored the adapter frameworks AdapterHub and MAD-X and work on argument mining.
• Graham Neubig (Carnegie Mellon University) has papers on probing what language models know, joint text–table pretraining and entity recognition for African languages.
• Kyunghyun Cho (New York University) worked on large-vocabulary and multilingual neural translation and on sentence representations.
• Philipp Koehn (Johns Hopkins University) shaped statistical machine translation with factored models, clause reordering and compound splitting.
• Yoav Goldberg (Bar-Ilan University) contributed to syntactic probing of LSTMs, faithful interpretability and removing protected attributes from representations.
• Preslav Nakov (Mohamed bin Zayed University of Artificial Intelligence) co-organised a long series of SemEval tasks on sentiment in Twitter and on offensive language.

Other names that recur among the most cited works include Bo Pang and Lillian Lee for sentiment analysis, Michael Collins for statistical parsing, Barbara J. Grosz for discourse theory, Samuel R. Bowman for benchmarks and sentence generation, Danqi Chen for open-domain QA and few-shot fine-tuning, and Jason Weston for dialogue and question answering.

Institutions and places

The field's centre of gravity is in the United States. Carnegie Mellon University, Stanford University, the University of Washington and New York University are the leading academic homes of authors on this shelf, joined by MIT, Johns Hopkins University, Princeton University and the University of Illinois Urbana-Champaign. Industry laboratories are unusually prominent: Google, Microsoft, Google DeepMind, Amazon, Meta and the nonprofit Allen Institute for Artificial Intelligence all employ many leading authors.

In Europe the University of Edinburgh has long been a stronghold, especially for translation, summarization and semantics. In China, Tsinghua University, Peking University and the Chinese Academy of Sciences are major centres. Bar-Ilan University in Israel and Mohamed bin Zayed University of Artificial Intelligence in the United Arab Emirates stand out for Hebrew and Arabic NLP respectively, reflecting this library's strong collection on Semitic languages.

The main venues are those of the Association for Computational Linguistics: its annual meeting, EMNLP and NAACL, the journal Computational Linguistics and Transactions of the ACL, plus many workshops and SemEval shared tasks. Most papers are open access through the ACL Anthology, which is one reason the field's literature is so complete here.

Open questions

• Meaning and grounding. Do language models that learn only from text truly understand it, and how much does grounding in perception or action add? Work in Vision & language — from LXMERT to studies of object hallucination in vision-language models — partly addresses this.
• Factuality. Generators produce fluent text that may be false. Measuring and reducing hallucination, as in TruthfulQA and FActScore, remains unsolved.
• Reasoning. Whether models reason or pattern-match, and how to make multi-step inference reliable; see Reasoning.
• Evaluation. Benchmarks saturate quickly and may leak into training data; reference-based metrics correlate poorly with human judgement for open generation.
• Bias, fairness and safety. Models absorb social biases from data, as shown in StereoSet, RealToxicityPrompts and racial bias in hate-speech data; see Bias, fairness & safety.
• Languages beyond English. Most resources still favour a handful of languages. How Multilingual is Multilingual BERT? and cross-lingual transfer of monolingual models show how far transfer goes; see Multilingual & cross-lingual NLP and Low-resource languages.
• Efficiency. Training and serving large models is costly in energy and money; see Efficient models & compression.
• Interpretability. Explaining model decisions faithfully, beyond attention maps; see Interpretability.

Neighbouring topics

NLP overlaps most with Language and Linguistics, which supplies its theories and many of its resources, and with the Machine translation shelf, historically the field's first application. It draws its learning methods from Artificial Intelligence and its hardware from Computer Science. Applied strands reach into Law through Legal AI, into History through work on ancient and historical languages, into Medicine through biomedical text mining such as SciBERT and PubMedQA, and into Sociology and Political Science and Education.

Within the shelf, the largest topics are machine translation and language models, followed by a distinctive cluster on Semitic languages and Hebrew & Aramaic, then evaluation, multilingual NLP, sentiment and morphology. Smaller but active topics include Speech & spoken language, Psycholinguistics & language acquisition, Historical linguistics & language change, Grammatical error correction & learner language, Idioms & multiword expressions and Code & programming languages.

Key works in this library

• A stochastic parts program — early proof that probabilistic tagging works on real text.
• Attention, Intentions, and the Structure of Discourse — a three-part theory of discourse structure.
• Assessing agreement: the kappa statistic — the case for chance-corrected agreement in annotation.
• A maximum entropy approach to NLP — log-linear models for language.
• Automatic Retrieval and Clustering of Similar Words — a thesaurus built from parsed text.
• Thumbs up? — supervised sentiment classification of reviews.
• Automatic Labeling of Semantic Roles — the first statistical role labeller.
• Accurate Unlexicalized Parsing — careful category splitting rivals lexicalised parsers.
• Head-Driven Statistical Models for Natural Language Parsing — the classic lexicalised parser.
• Non-local information in extraction via Gibbs sampling — long-distance constraints in entity recognition.
• The Proposition Bank — semantic roles over the Penn Treebank.
• Domain Adaptation with Structural Correspondence Learning — transferring models to new domains via pivot features.
• Hierarchical Phrase-Based Translation — synchronous grammars for statistical translation.
• Linguistic Regularities in Continuous Space Word Representations — word vectors capture analogies as offsets.
• SQuAD — the reading comprehension benchmark.
• Diachronic word embeddings and laws of semantic change — embeddings over time trace meaning shift.
• GLUE, a multi-task understanding benchmark — one leaderboard for sentence understanding.
• Energy and policy considerations for deep NLP — the carbon and money cost of training.
• A Primer in BERTology — a survey of what probing found inside BERT.
• BART: denoising sequence-to-sequence pretraining — denoising pretraining for generation and understanding.
• Transformers: State-of-the-Art Natural Language Processing — the shared library of pretrained models.
• Rethinking the role of demonstrations — what in-context examples actually contribute.
• TruthfulQA: Measuring How Models Mimic Human Falsehoods — testing whether models repeat popular misconceptions.
• FActScore — scoring long generations fact by fact.