🏔️ Large language models

About 16 minutes to read · every paper named here has its own page in this library

What it is

A language model assigns probabilities to sequences of words. Given the beginning of a sentence, it estimates which word, or piece of a word, is likely to come next. A large language model applies the same idea at enormous scale. It has billions or hundreds of billions of adjustable parameters. It trains on a large share of the public written web, plus books and code. The resulting system can be prompted in ordinary language to translate, summarize, explain, argue, write programs or hold a conversation. Most users meet one through a chatbot interface.

"Large" has no fixed threshold, and it has moved over time. GPT-3 had 175 billion parameters, roughly ten times any earlier dense language model, while GPT-2 had 1.5 billion (Brown et al. 2020). Some overviews limit their scope to pretrained models of about 10 billion parameters and above (Naveed et al. 2023). A more useful criterion looks at behaviour and use. One major survey separates LLMs from earlier pretrained models on three grounds (Zhao et al. 2023):

• they show abilities such as in-context learning;
• people use them mainly through prompting rather than task-specific fine-tuning;
• building them blurs the line between research and engineering.

That survey places LLMs as the fourth stage of language modelling. Statistical n-gram models came first, then neural models, then pretrained models such as ELMo and BERT (Zhao et al. 2023).

The Stanford report on "foundation models" gave the field much of its vocabulary (Bommasani et al. 2021). In this view, one model is pretrained with self-supervision on broad data and then adapted to many tasks. The report stresses two consequences. The first is emergence: capabilities such as in-context learning arise from scale rather than explicit design. A narrower, threshold-based sense of the word is discussed under History. The second is homogenization: many applications rest on a few base models, so their strengths and their flaws spread together (Bommasani et al. 2021). For natural language processing and deep learning, the practical result is that one pretrained network now underlies tasks that once each needed a separately trained system (Bommasani et al. 2021).

How it works

Next-token prediction

Text is first cut into tokens by subword tokenization. Common tokenizers include byte-pair encoding (BPE), WordPiece and unigram models (Naveed et al. 2023). LLaMA, for example, uses BPE as implemented in SentencePiece. It splits numbers into single digits and falls back to bytes for unknown characters (Touvron et al. 2023). The model then learns to predict each token from the tokens before it. The standard training loss is the average negative log-likelihood of the corpus:

L(θ) = − (1/T) Σ_{t=1..T} log p_θ(x_t | x_1, …, x_{t−1})

Here x_1 … x_T is a training sequence of T tokens, and θ stands for all the parameters of the network. The term p_θ(x_t | x_1, …, x_{t−1}) is the probability the model assigns to the true next token given everything before it. Minimising L(θ) is the same as minimising cross-entropy between the model's predictions and the data. The scaling-law literature uses this same quantity to measure quality (Kaplan et al. 2020). Because the labels come from the text itself, no human annotation is needed. This self-supervision is what makes web-scale training possible (Bommasani et al. 2021).

At generation time, the model samples one token, appends it to the input and repeats. This autoregressive loop is simple. Bubeck and colleagues argue that it also causes some weaknesses, such as poor planning, because the model cannot revise what it has already written (Bubeck et al. 2023).

The decoder-only transformer

Almost all current LLMs are built on the transformer of Vaswani et al. (2017). Its core operation, self-attention, is usually written as:

Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V

Q, K and V are matrices of query, key and value vectors, computed from the token representations by learned projections. d_k is the dimension of the keys, and softmax turns each row of scores into weights that sum to one. In a decoder-only model, a causal mask stops each position from attending to later positions, so the network can only use the past. Surveys group pretrained models into three families (Pan et al. 2023; Chua; & Li 2024):

• encoder-only, such as BERT;
• encoder-decoder, such as T5;
• decoder-only, such as GPT, PaLM and LLaMA.

The decoder-only family dominates generative LLMs. PaLM is a dense decoder-only transformer (Chowdhery et al. 2022). LLaMA adds now-standard refinements (Touvron et al. 2023):

• RMSNorm, which rescales activations by their root mean square, applied before each sub-layer;
• the SwiGLU activation, a gated form of the feed-forward nonlinearity;
• rotary positional embeddings.

Pretraining on web-scale text

Pretraining corpora mix filtered web crawls with higher-quality sources. About two-thirds of LLaMA's corpus is CommonCrawl, cleaned with CCNet, a pipeline that deduplicates web pages and filters them by language and quality. It adds C4, GitHub, Wikipedia, books, arXiv and Stack Exchange, for roughly 1.4 trillion tokens (Touvron et al. 2023). PaLM trained on 780 billion tokens (Chowdhery et al. 2022). GPT-4 used public internet data and licensed third-party data (OpenAI et al. 2023).

How to spend a training budget is governed by scaling laws. Kaplan and colleagues found that test loss falls as a power law in each of three quantities, as long as the other two are not the bottleneck (Kaplan et al. 2020):

L(N) ∝ N^(−α_N),   L(D) ∝ D^(−α_D),   L(C) ∝ C^(−α_C)

N is the number of non-embedding parameters, D the number of training tokens and C the training compute. The α are fitted exponents. In their experiments they were about 0.076 for parameters and 0.095 for data. For compute allocated optimally, the exponent was about 0.050 (Kaplan et al. 2020). Network shape, meaning depth, width and number of heads, mattered little. Their compute rule favoured very large models trained for relatively few tokens (Kaplan et al. 2020).

DeepMind's Chinchilla study revised that rule. Using over 400 training runs, it concluded that parameters and tokens should grow in equal proportion (Hoffmann et al. 2022). Chinchilla has 70 billion parameters and was trained on 1.4 trillion tokens. It used the same compute as the 280-billion-parameter Gopher and beat it (Hoffmann et al. 2022). LLaMA went further. It deliberately trained past the Chinchilla optimum, because inference cost dominates once a model is deployed (Touvron et al. 2023).

From base model to assistant

A pretrained model continues text. It does not, by default, follow requests. Two further stages turn it into an assistant.

The first is supervised instruction tuning, also called supervised fine-tuning (SFT). In InstructGPT, 40 hired labelers wrote demonstration answers to real prompts, and GPT-3 was fine-tuned on them (Ouyang et al. 2022).

The second is reinforcement learning from human feedback (RLHF). Labelers rank several model outputs, and a reward model learns those preferences. The policy is then optimised against the reward model with proximal policy optimisation (PPO), a standard reinforcement-learning algorithm (Ouyang et al. 2022). Raters preferred the outputs of a 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3 (Ouyang et al. 2022). Bai and colleagues applied the same recipe to dialogue assistants meant to be both helpful and harmless. They found that, for large models, alignment training cost little or nothing on standard benchmarks (Bai et al. 2022).

More recently, reinforcement learning with verifiable rewards has been used to teach reasoning. DeepSeek-R1-Zero was trained with no supervised warm-up. Its algorithm was group relative policy optimisation (GRPO), which rates each sampled answer against the others drawn for the same question instead of relying on a critic model. Rule-based rewards checked the final answers. The journal version of the paper tracks pass@1 on the 2024 American Invitational Mathematics Examination (AIME) during training. Pass@1 is the share of problems solved by a single sampled answer. It rose from 15.6% to 77.9% (DeepSeek-AI et al. 2025).

History

Pretraining before "large"

The pretrain-then-adapt paradigm took hold in 2018. Radford et al. (2018) introduced GPT, a generatively pretrained transformer. Devlin et al. (2018) introduced BERT, trained with masked language modelling. The foundation-models report dates the current era to about 2018–2019, starting with BERT. It contrasts BERT's masked objective with GPT's next-word prediction (Bommasani et al. 2021). Three enablers came together at that point (Bommasani et al. 2021):

• GPU throughput and memory grew roughly tenfold in four years;
• transformers parallelised well;
• self-supervised objectives could exploit unlabelled text.

GPT-3 and in-context learning

GPT-3 marked the shift to prompting. The model was evaluated with no weight updates. The task description and a few examples were simply placed in the prompt, a mechanism its authors named in-context learning (Brown et al. 2020). Few-shot performance grew faster with model size than zero-shot performance. In some settings it approached fine-tuned systems (Brown et al. 2020). This made few-shot learning through prompts a central research object.

PaLM, Chinchilla and emergent abilities

In 2022 Google trained PaLM, a 540-billion-parameter model, on 6,144 TPU v4 chips. It reached 46.2% model FLOPs utilisation, the share of the chips' peak arithmetic rate spent on the model's own computation (Chowdhery et al. 2022). It set new few-shot results on 28 of 29 widely used English benchmarks. On some BIG-bench tasks, quality jumped only at the largest size (Chowdhery et al. 2022).

Such jumps were collected under the label "emergent abilities". These are skills that small models lack and larger models suddenly show, so they cannot be extrapolated from small-scale trends (Wei et al. 2022). For example, GPT-3's accuracy on multi-digit arithmetic stayed near zero until about 2·10²² training FLOPs, around 13 billion parameters (Wei et al. 2022). Chinchilla's revision of the scaling rule, covered under pretraining above, came out the same year (Hoffmann et al. 2022).

ChatGPT and GPT-4

ChatGPT, released at the end of 2022, made the instruction-tuned and RLHF-trained assistant a mass-market product. It descends from the InstructGPT line of work (Ouyang et al. 2022). Research output followed. After ChatGPT's release, arXiv papers mentioning "large language model" in the title or abstract went from 0.40 to 8.58 per day (Zhao et al. 2023). Early studies showed ChatGPT near the roughly 60% pass mark on the US medical licensing exam (Kung et al. 2023). Other work showed it outperforming crowd-workers on text annotation (Gilardi et al. 2023).

GPT-4 (2023) accepts images as well as text. Its report withholds architecture, size and compute details (OpenAI et al. 2023). It scored about the 90th percentile on a simulated bar exam. Its final loss was forecast in advance from runs using up to 10,000 times less compute (OpenAI et al. 2023). This does not contradict emergence. Aggregate loss can fall smoothly and predictably even when particular abilities appear abruptly (OpenAI et al. 2023; Wei et al. 2022). A Microsoft team with early access argued that GPT-4 showed "sparks" of general intelligence. Their study was purely observational (Bubeck et al. 2023).

Open models

Meta's LLaMA (2023) showed that competitive models could be trained only on public data. LLaMA-13B outperformed GPT-3 on most benchmarks, and the weights were shared with researchers (Touvron et al. 2023). Llama 2 added fine-tuned chat models (Touvron et al. 2023). The Llama 3 family followed (Grattafiori et al. 2024), along with specialised offshoots such as Code Llama (Rozière et al. 2024). Openly released reasoning models such as DeepSeek-R1, together with smaller models distilled from it, extended the trend (DeepSeek-AI et al. 2025).

Methods

Prompting and reasoning

Prompt engineering became a method in its own right. Reynolds and McDonell argued that few-shot examples mostly help GPT-3 locate a task it already knows rather than learn a new one. In their translation case study, well-written zero-shot prompts matched or beat few-shot prompts (Reynolds & McDonell 2021).

Chain-of-thought prompting adds worked intermediate steps to each example. With PaLM 540B it raised accuracy on GSM8K, a set of grade-school maths word problems, from about 18% to about 57%. The benefit appeared only above roughly 100 billion parameters (Wei et al. 2022).

Kojima and colleagues showed that a single trigger, "Let's think step by step", works with no examples at all. It raised InstructGPT's accuracy on MultiArith, a set of multi-step arithmetic word problems, from 17.7% to 78.7% (Kojima et al. 2022).

Least-to-most prompting breaks a problem into easier subquestions and answers them in sequence. It generalised to problems harder than its examples. On every split of SCAN, a benchmark that maps simple commands to action sequences, it reached at least 99% (al. 2022).

Tree of Thoughts extends the idea to deliberate search over alternative reasoning paths (Yao et al. 2023). These lines of work sit within the wider study of reasoning.

Efficient adaptation and serving

Full fine-tuning of a 175-billion-parameter model is costly. LoRA freezes the pretrained weights and learns a low-rank update in selected layers. For GPT-3 it cut trainable parameters by about 10,000 times and GPU memory by about three times, with no added inference latency (Hu et al. 2021). QLoRA combines this kind of adapter with a quantised base model so that fine-tuning fits on far smaller hardware (Dettmers et al. 2023). Overviews also list pruning, quantization, distillation and context-length interpolation as efficiency tools (Naveed et al. 2023). More on this is under efficient models.

Serving is its own bottleneck. Each request's key-value cache grows as tokens are generated, and its final length is not known in advance. Existing systems therefore reserved space for the maximum possible length. Only 20.4–38.2% of their cache memory ended up storing actual token states (Kwon et al. 2023). PagedAttention borrows paged virtual memory from operating systems and allocates fixed-size blocks only when needed. The resulting vLLM system delivers 2–4 times the throughput of earlier systems at the same latency (Kwon et al. 2023). Hardware concerns are covered further under GPU and parallel computing.

Retrieval, tools and agents

Retrieval-augmented generation lets a model consult an external document store before answering. It addresses stale knowledge and untraceable claims. A survey sorts these systems into Naive, Advanced and Modular RAG (Gao et al. 2023). Another groups retrieval-augmented LLMs by architecture, training strategy and application (Chua; & Li 2024). Combining LLMs with a knowledge graph is a related programme, with knowledge flowing in either direction (Pan et al. 2023).

LLMs also serve as the controller of autonomous agents. Surveys describe such agents in terms of modules for profile, memory, planning and action (Wang et al. 2023), or brain, perception and action (Xi et al. 2023). One concrete case is Coscientist, a GPT-4-based agent that planned and ran chemistry experiments through web search, code execution and laboratory hardware (Boiko et al. 2023).

Code and multimodality

Training on source code produced Codex. It was fine-tuned on public GitHub code, starting from GPT models of up to 12 billion parameters (Chen et al. 2021). Code Llama continued Llama 2 pretraining on 500 billion tokens of code-heavy data (Rozière et al. 2024). See also code and programming.

Among multimodal models, several systems connect a vision encoder to a language model:

• BLIP-2 links a frozen image encoder to a frozen LLM through a small Querying Transformer (Q-Former). Its learned query vectors pick out the visual features relevant to text (Li et al. 2023).
• InstructBLIP feeds the instruction itself into that connector (Dai et al. 2023).
• LLaVA trains on image-grounded instruction data written by a text-only GPT-4 (Liu et al. 2023). Its successor LLaVA-1.5 showed that a simple MLP connector, a small multi-layer perceptron, suffices (Liu et al. 2023).
• MiniGPT-4 trains only a single linear layer between BLIP-2's visual stack and Vicuna (Zhu et al. 2023).

A survey maps these designs as encoder, connector and LLM (Yin et al. 2023).

Datasets and benchmarks

Evaluation and benchmarks for LLMs have become a field of their own. A survey organises the field around three questions: what abilities to test, on which benchmarks, and with which protocols. It finds reasoning and robustness to perturbed inputs to be recurring weak points (Chang et al. 2023). The main benchmark types are:

Code evaluation shows why the scoring method matters. Text-overlap metrics such as BLEU often gave wrong programs higher scores than correct ones. HumanEval instead runs unit tests on 164 hand-written Python problems. Codex-12B solved 28.8% of them with a single sample (Chen et al. 2021). MBPP contributed 974 entry-level programming tasks (Austin et al. 2021).

In medicine, MultiMedQA combined existing question-answering sets with a new collection of consumer health queries. Flan-PaLM reached 67.6% on MedQA, but physicians found serious problems with its long-form answers (Singhal et al. 2022).

Two trends stand out. First, models are increasingly used as judges. G-Eval scores generated text with GPT-4 (Liu et al. 2023), and MT-Bench and Chatbot Arena study when an LLM judge agrees with people (Zheng et al. 2023). Second, test-set contamination is a constant worry:

• GPT-3's authors flagged it (Brown et al. 2020);
• GPT-4's report removed contaminated exam questions (OpenAI et al. 2023);
• the evaluation survey calls for dynamic tests that resist leakage (Chang et al. 2023).

People and institutions

A handful of research programmes shaped the methods above. Jason Wei led the work on chain-of-thought prompting (Wei et al. 2022) and on emergent abilities (Wei et al. 2022). He also co-authored the clinical-knowledge study (Singhal et al. 2022). Denny Zhou, at Google, co-authored chain-of-thought and emergence and led least-to-most prompting (al. 2022). Quoc Le co-authored chain-of-thought and the study of program synthesis with large models (Austin et al. 2021). Colin Raffel, at the University of Toronto, the Vector Institute and Hugging Face, co-authored the emergent-abilities paper (Wei et al. 2022).

Efficiency and open infrastructure have their own figures. Luke Zettlemoyer, at the University of Washington and Meta, co-authored QLoRA (Dettmers et al. 2023). Ion Stoica, at the University of California, Berkeley, belongs to the group behind PagedAttention and vLLM (Kwon et al. 2023). The same group produced the Chatbot Arena work (Zheng et al. 2023).

Surveys have organised the field's knowledge. Yue Zhang, at Westlake University, co-authored the survey on evaluating LLMs (Chang et al. 2023). Ji-Rong Wen, at Renmin University of China, belongs to the group behind a widely used LLM survey (Zhao et al. 2023).

Industrial laboratories built most frontier models: OpenAI (GPT-3, InstructGPT, GPT-4), Google (PaLM), DeepMind (Chinchilla), Meta (LLaMA) and DeepSeek (DeepSeek-R1). Academia supplies much of the analysis, evaluation and open tooling. Active universities include:

• Carnegie Mellon University
• Stanford University
• Tsinghua University
• Peking University
• Zhejiang University
• the University of Washington
• the University of California, Berkeley

They work alongside the Chinese Academy of Sciences, Microsoft, Google and Alibaba Group.

Open questions

Hallucination and factuality

LLMs produce fluent statements that are false or unsupported. One survey divides the problem into two kinds. Factuality hallucinations conflict with verifiable facts. Faithfulness hallucinations diverge from the input or from the model's own earlier text. The survey traces causes to data, training and decoding (Huang et al. 2023). Another proposes three types instead: input-conflicting, context-conflicting and fact-conflicting (Zhang et al. 2023).

The stakes are concrete. One cited study found leading models hallucinating in 69% to 88% of answers to specific legal queries (Chua; & Li 2024). RLHF reduced fabrication on closed-domain tasks from 41% to 21% of cases for InstructGPT, which still leaves many errors (Ouyang et al. 2022). Retrieval helps, but retrieval-augmented systems still hallucinate (Huang et al. 2023).

Bias, safety and security

Models trained on unfiltered web text absorb social bias and can amplify it. A survey classifies bias metrics by whether they inspect embeddings, token probabilities or generated text. It sorts mitigations by pipeline stage, from pre-processing to post-processing (Gallegos et al. 2023). RLHF did not noticeably reduce InstructGPT's bias scores on two tests (Ouyang et al. 2022). Winogender probes gendered pronoun resolution, and CrowS-Pairs compares paired stereotyped and less-stereotyped sentences.

On security, a review of 281 publications separates defensive uses, offensive uses and the models' own vulnerabilities. Those vulnerabilities include data poisoning, extraction of training data and prompt injection (Yao et al. 2023). Medical and educational deployment raise their own concerns. In medicine, models can give potentially harmful clinical answers (Singhal et al. 2022). In education, the concerns are the need for AI literacy, fact-checking and fair access (Kasneci et al. 2023). These issues are gathered under bias, fairness and safety.

Do they reason, and why do abilities emerge?

Whether generated chains of thought reflect real reasoning remains unsettled. The chain-of-thought authors note that the steps can be wrong even when the answer is right (Wei et al. 2022). The emergence paper documents sharp thresholds but does not explain them (Wei et al. 2022). The prompt-programming critique, described under Prompting and reasoning, adds a further doubt. Some apparent in-context learning may be task location rather than learning (Reynolds & McDonell 2021). Explaining what models represent internally is the agenda of interpretability.

Cost, openness and coverage

Training remains expensive. LLaMA-65B took about 21 days on 2,048 A100 GPUs (Touvron et al. 2023), and frontier reports increasingly withhold such details (OpenAI et al. 2023). Three lines of work lower the bill without removing it:

• compute-optimal training (Hoffmann et al. 2022);
• parameter-efficient tuning (Hu et al. 2021);
• better serving (Kwon et al. 2023).

Finally, much LLM research centres on English, so speakers of other languages may benefit less (Kasneci et al. 2023). That gap links LLM research to work on multilingual NLP and low-resource languages.

The library in figures