🗣️ Language model
A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into next-token predictions, so it can score how likely a text is and generate new text one token at a time.
What it is
A language model assigns a probability to any string of text. That single ability serves two purposes. It can rank alternatives, for example deciding which of several acoustically similar transcriptions a speech recognizer should prefer. It can also generate text, by repeatedly sampling a likely continuation.
A classic early application was speech recognition, where a decoder needs a prediction of the next word given the words already heard (Jelinek & Lafferty 1991).
Over time the term has widened. It now also covers neural networks pretrained on huge text collections and then adapted to tasks that have little to do with probability scoring as such. These include question answering, coding, medical exams and tool use.
A useful distinction is between autoregressive models and masked models. Autoregressive models predict each token from the tokens to its left. Masked models, such as BERT, predict hidden tokens from context on both sides (Devlin et al. 2019). Only the first kind defines a proper distribution over whole sequences in the textbook sense. Both are nevertheless called language models in current usage.
The key idea
The probability of a sequence is decomposed with the chain rule of probability, and quality is measured with perplexity:
P(w1, …, wn) = ∏ᵢ P(wi | w1, …, wi−1)
PPL = exp( −(1/n) · Σᵢ log P(wi | w1, …, wi−1) )
Here w1 … wn are the tokens of a text of length n. The notation wi is the i-th token, and w1 … wi−1 (often written w<i) is everything before it. The product ∏ᵢ and the sum Σᵢ run over positions i = 1 … n. P(wi | w<i) is the model's predicted probability of the actual next token, and log is the natural logarithm.
PPL is perplexity, the exponentiated average negative log-likelihood per token on held-out text. Intuitively, it is the effective number of equally likely choices the model is weighing at each step, so lower is better. Per-token perplexity is directly comparable only between models that share the same tokenization and test set. Normalizing the log-likelihood per character or per byte (for example, bits per byte) allows comparison across different tokenizers.
The chain rule is exact but leaves the conditional P(wi | w<i) to be estimated. The history of language modeling is largely a history of how that conditional is approximated:
- n-gram models make a Markov assumption: only the last n−1 tokens matter. Probabilities are then estimated from counts in a corpus.
- Smoothing refines this estimation, because most long n-grams never occur in training. Without it, an unseen n-gram gets probability zero, and if such an n-gram appears in the test text, perplexity becomes infinite. Classic remedies include add-one (Laplace) smoothing, Good–Turing estimation, Katz back-off (Katz 1987) and Kneser–Ney smoothing (Kneser and Ney 1995). All of them move probability mass from observed to unobserved events or fall back to shorter contexts.
- Grammar-based models replace counts with a probabilistic grammar. They then need prefix probabilities, which Jelinek and Lafferty showed can be computed in time cubic in the prefix length (Jelinek & Lafferty 1991).
- Neural models compute the conditional with a learned function of embedded tokens. Similar words receive similar vectors, so probability generalizes across contexts never seen verbatim (Bengio et al. 2003).
History and founding papers
The idea of modeling text as a stochastic source goes back to Shannon (1948). The statistical, count-based tradition was developed largely in speech recognition research at IBM in the 1970s and 1980s, associated with Jelinek and colleagues.
A parallel line of work, also motivated by speech recognition, used probabilistic grammars instead of counts. Jelinek and Lafferty derived a left-to-right variant of the inside algorithm. With it, a stochastic context-free grammar can serve as a predictive language model that updates as each new word arrives (Jelinek & Lafferty 1991).
An influential neural probabilistic model came from Bengio and colleagues. They proposed learning word representations jointly with a neural network that outputs next-word probabilities (Bengio et al. 2003).
A later survey organizes the whole field into four stages: statistical models, neural models, pretrained models and large models (Zhao et al. 2023).
| Stage | Typical method | Example |
|---|---|---|
| Statistical | n-gram counts with smoothing | back-off trigram models |
| Neural | learned embeddings, neural predictor | NPLM (Bengio et al. 2003) |
| Pretrained | self-supervised pretraining, then fine-tuning | BERT (Devlin et al. 2019) |
| Large | very large models with emergent abilities, used mainly through prompting | GPT-3 (Brown et al. 2020) |
The survey separates the last stage from earlier pretrained models on three grounds. Large models show emergent abilities such as in-context learning. They are used chiefly through prompts instead of fine-tuning. They are also much bigger: GPT-3 (175B parameters) and PaLM (540B) are contrasted with BERT (330M) (Zhao et al. 2023).
Pretraining made the language-modeling objective a general source of linguistic knowledge. BERT pretrained deep bidirectional Transformers and then fine-tuned them for understanding tasks (Devlin et al. 2019).
Probing with cloze statements showed that such models can recall many relational facts without any fine-tuning. On open-domain questions, BERT-large reached 57.1% precision@10, against 63.5% for a supervised, task-specific baseline (Petroni et al. 2019).
Scaling then became central. Test loss was found to follow predictable power-law trends in model size, data and compute (Kaplan et al. 2020). A very large autoregressive model could perform new tasks from a few examples placed in its prompt (Brown et al. 2020). Later systems such as PaLM, with 540B parameters, pushed scale further (Chowdhery et al. 2022; Zhao et al. 2023).
Some abilities do not improve smoothly with scale. They stay near chance and then jump once models pass a certain size. This does not contradict the smooth loss trends. Those trends describe average test loss, whereas emergence concerns specific downstream tasks whose performance cannot be forecast by extrapolating from smaller models. For GPT-3, for example, accuracy on BIG-Bench arithmetic (three-digit addition and subtraction, two-digit multiplication) remained close to zero until about 2·10²² training FLOPs, roughly 13B parameters (Wei et al. 2022).
How it is used today
Most current language models are pretrained once and then adapted.
Domain adaptation can mean continued pretraining of a general model or pretraining from scratch on specialized text. SciBERT keeps the BERT-Base architecture but is pretrained on about 3.17B tokens of scientific papers, with its own in-domain vocabulary. When fine-tuned, it beat BERT-Base by an average of 2.11 F1. That average covers entity recognition, PICO extraction, classification of texts and of relations, and dependency parsing, on datasets from biomedicine, computer science and mixed domains (Beltagy et al. 2019). PubMedBERT, trained from scratch purely on biomedical text, outperformed mixed-domain models built by continual pretraining (Gao; & Poon 2020).
Parameter-efficient methods such as low-rank adaptation make fine-tuning large models cheaper (Hu et al. 2021). Instruction tuning and alignment methods teach models to follow instructions:
- reinforcement learning from human feedback, applied to instruction following (Ouyang et al. 2022);
- supervised fine-tuning on a large, scaled-up collection of instruction-formatted tasks (Chung et al. 2022);
- direct preference optimization, which skips an explicit reward model (Rafailov et al. 2023).
Foundation models released to the research community, such as LLaMA, broadened access for research (Touvron et al. 2023).
Prompting has become a method in its own right:
- With chain-of-thought exemplars, PaLM 540B scored about 57% on the GSM8K math benchmark, versus about 18% with standard prompting (Wei et al. 2022).
- Simply appending a step-by-step trigger raised InstructGPT's GSM8K accuracy from 10.4% to 40.7% with no examples at all (Kojima et al. 2022).
- In a separate study, sampling many reasoning paths and taking a majority vote lifted PaLM-540B's GSM8K accuracy from 56.5% with greedy chain-of-thought decoding to 74.4% (Wang et al. 2022).
Interleaving reasoning with actions such as search calls turns a language model into an agent (Yao et al. 2022). A broad literature now treats LLMs as controllers built from profiling, memory, planning and action modules (Wang et al. 2023).
Applications span many fields:
- Code: a GPT model fine-tuned on code, Codex-12B, solved 28.8% of HumanEval problems at pass@1, that is, with a single sample (Chen et al. 2021).
- Medicine: Flan-PaLM reached 67.6% on MedQA (Singhal et al. 2022). ChatGPT performed at or near the roughly 60% passing mark on 350 text-only items from the June 2022 public USMLE sample set, covering Steps 1, 2CK and 3, after image-based items were removed (Kung et al. 2023).
- Education: language models raise both opportunities and risks for teaching and learning (Kasneci et al. 2023).
- Vision-language systems: a frozen language model can be connected to a frozen image encoder through a small trainable bridge (Li et al. 2023).
- Serving at scale: efficient deployment depends on systems work such as paged management of attention memory (Kwon et al. 2023).
The growth of the field is visible in publication rates. After ChatGPT's release, the daily rate of arXiv papers with "large language model" in their title or abstract rose from 0.40 to 8.58 (Zhao et al. 2023).
Open questions
Factual reliability remains a prominent weakness. Models produce fluent but unsupported content. A recent taxonomy separates conflicts with real-world facts from divergence from the input, and traces causes to data, training and decoding (Huang et al. 2023).
Long contexts are not used evenly. Accuracy peaks when the relevant evidence opens or closes the input, and falls when that evidence is buried in the middle (Liu et al. 2023).
Evaluation itself is unsettled. Perplexity measures prediction, not usefulness. Benchmark scores face training-data contamination and sensitivity to prompt wording, and models still show weaknesses in reasoning and robustness (Chang et al. 2023).
Calibration is also imperfect. Weighting sampled answers by their length-normalized probability performs about as well as plain voting. Weighting by unnormalized probability does clearly worse, which suggests the model's probabilities are poorly calibrated (Wang et al. 2022).
It is debated whether generated reasoning reflects genuine reasoning, since chains can be wrong even when answers are right (Wei et al. 2022). It also remains unexplained why some abilities emerge abruptly with scale (Wei et al. 2022).
Finally, the research focus on English risks leaving speakers of other languages with weaker tools (Kasneci et al. 2023). In high-stakes domains, safety and bias evaluations are still incomplete (Singhal et al. 2022).
Key papers
Related concepts
- Transformer and attention — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
- Scaling laws — Scaling laws are empirical relationships, usually power laws, that describe how a neural network's performance, often its test loss…
- Subword tokenization — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…
- Word embeddings — Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are…
More papers
Topics: 🧠 Language models & pretraining
Shelves: 💬 Natural language processing · 🤖 AI & machine learning
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/language-model/