✂️ Subword tokenization
Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of fixed size is learned from a corpus, so frequent words stay whole and rare or unseen words break into known pieces.
What it is
A neural text model can only take as input a finite set of symbols. Word-level vocabularies leave rare and unseen words out, and they grow very large in morphologically rich languages. Character or byte sequences avoid unknown symbols, but they make inputs long and expensive to process. Subword tokenization sits between these two options.
A tokenizer has two parts:
- a learned inventory of pieces;
- a procedure that maps any string onto a sequence of those pieces.
Common words usually become single tokens. Rarer words become a few fragments. Depending on the vocabulary, a word like "unhappiness" might come out as "un", "happi" and "ness".
The approach became standard in neural machine translation. It was then carried into pretrained encoders and large language models (Sennrich et al. 2016; Mielke et al. 2021). Tokenizers are commonly compared by fertility (the average number of tokens per word), by compression, and above all by how well the resulting model performs on later tasks. The first two are convenient proxies, but how well they predict model quality is disputed (Ali et al. 2024; Schmidt et al. 2024).
The key idea
Most subword vocabularies are built in one of two ways. Byte-pair encoding grows a vocabulary bottom-up by merging symbols. The unigram language model starts large and prunes top-down.
BPE merge step: (a*, b*) = argmax over adjacent pairs (a, b) of count(a, b)
replace every occurrence of a b with the new symbol ab
Unigram LM: P(x) = p(x1) · p(x2) · … · p(xM)
x* = argmax over x in S(X) of P(x)
In the BPE lines, a and b are adjacent symbols in the training corpus, and count(a, b) is how often they occur side by side. The pair with the highest count, (a*, b*), becomes a new symbol ab.
In the unigram lines, X is a word or sentence and S(X) is the set of all ways to segment X with the current vocabulary. Each x is one such segmentation, made of pieces x1 to xM. The value p(xi) is the probability of piece xi, and x* is the most probable segmentation.
Byte-pair encoding works step by step:
- Split every word into characters, with an end-of-word marker.
- Count all adjacent symbol pairs.
- Merge the most frequent pair everywhere and record that merge.
- Repeat until a chosen number of merges has been performed.
To tokenize new text, the recorded merges are replayed in the order they were learned.
WordPiece, attributed to Schuster and Nakajima (2012), is similar to BPE. The difference is that it picks the merge that most raises the likelihood of the training data, rather than the most frequent raw pair. In the unigram model, piece probabilities are fitted with EM (expectation–maximization). The best segmentation is found with Viterbi decoding, a dynamic-programming search for the highest-scoring path. This also makes it possible to sample alternative segmentations during training. BPE-dropout later brought a similar kind of sampling to BPE (Kudo 2018; Provilkov et al. 2020).
| Method | Direction | Segmentation of new text |
|---|---|---|
| BPE | bottom-up merges | replay merge table |
| WordPiece | bottom-up, likelihood-driven | greedy longest match (BERT's inference procedure) |
| Unigram LM | top-down pruning | Viterbi over piece probabilities |
History and founding papers
Byte-pair encoding began as a data-compression method by Gage (1994). It was adapted to the translation of rare words (Sennrich et al. 2016), and this made it the default segmentation for neural machine translation (Mielke et al. 2021). In parallel, static word embeddings were extended to use subword information (Bojanowski et al. 2017).
Kudo then proposed the unigram language model together with subword regularization. In this scheme the model sees several sampled segmentations of the same word during training (Kudo 2018).
SentencePiece made both BPE and unigram training work directly on raw sentences (Kudo & Richardson 2018). It replaces whitespace with the meta-symbol ▁ (U+2581) and then handles it like any other character, so detokenization is lossless. Users set the final vocabulary size directly instead of a merge count. On the KFTT English–Japanese corpus, a shared 8k SentencePiece vocabulary reached 29.55 BLEU (an n-gram overlap score for translation) for ja→en. An 80k word-level baseline reached 28.24 BLEU on the same task (Kudo & Richardson 2018).
BPE-dropout later brought regularization back to plain BPE. During training it skips each candidate merge with probability p, typically 0.1. At inference, plain BPE is applied (p = 0). Across IWSLT, WMT14 and ASPEC translation directions, it beat standard BPE by up to 2.3 BLEU and Kudo's method by up to 0.9 BLEU. On clean WMT14 test sets, the gains over BPE were 0.45–1.5 BLEU. They rose to 1.58–2.32 BLEU when 10% of source words carried a one-character typo (Provilkov et al. 2020).
Byte-level BPE runs the merges over UTF-8 bytes, so no input is ever out of vocabulary. In translation it matched BPE quality with about one-eighth of the vocabulary (Wang et al. 2019).
A separate line of work comes from unsupervised morphology. Morfessor 2.0 re-implements Morfessor Baseline, the method of Creutz and Lagus (2002), which was first released as Morfessor 1.0 in 2005. It uses MAP (maximum a posteriori) estimation that balances a lexicon prior against data likelihood (Smit et al. 2014). Later models in this line include:
- Bayesian multilingual segmenters, which cut relative error by up to 24% over a monolingual counterpart (Snyder & Barzilay 2008);
- log-linear models with priors in the spirit of MDL (minimum description length). On the full Arabic Penn Treebank they reached 77.7 F1 (the harmonic mean of boundary precision and recall), against 74.9 for Morfessor Categories-MAP (Poon et al. 2009);
- adaptor grammars guided by about 1000 gold-segmented word types (Sirts & Goldwater 2013).
For Arabic, one system combined a morphological analyzer with SVM classifiers that choose among its candidate analyses. It reported a tokenization error of 0.7% on the Penn Arabic Treebank, against 2.9% for an earlier system (Habash & Rambow 2005). Joint lattice parsing for Turkish and Hebrew showed that the right segmentation can depend on syntax (Seeker & Çetinoğlu 2015).
How it is used today
Most pretrained language models ship with a subword tokenizer (Mielke et al. 2021). The shared WordPiece vocabulary of multilingual BERT has 119,547 tokens (Rust et al. 2021). ChatGPT and BLOOMZ both rely on byte-level BPE (Ahia et al. 2023). Pretrained BPE embeddings have been released for 275 languages. In English entity typing, an 11 MB BPE model matched FastText vectors of roughly 6 GB (Heinzerling & Strube 2018).
Vocabulary size is a central design choice. In 2.6B-parameter decoder experiments, about 33k entries sufficed for English. A five-language mix (English, German, French, Italian, Spanish) needed roughly three times as many. Reusing an English-centric tokenizer for that mix, instead of one trained on it, raised training compute by up to 68% (Ali et al. 2024).
For code models, vocabulary size correlated only weakly with accuracy on HumanEval, a benchmark of Python code-generation problems. The pre-tokenization regular expression mattered much more. A base model's tokenizer could be swapped with little loss once fine-tuning exceeded about 50B tokens (Dagan et al. 2024).
Swapping tokenizers also underlies cross-lingual transfer. Each new subword embedding can be initialized from semantically similar old ones. The similar tokens are found with multilingual static word embeddings that place both languages in one shared space. This matters because embeddings make up about 31% of the parameters of the base-size RoBERTa (Minixhofer et al. 2022).
Mismatched vocabularies complicate knowledge distillation. For example, the BLOOMZ tokenizer covers only 30.03% of Mistral's vocabulary (Boizard et al. 2024). Discrete tokenization in a broader sense, mapping signals to discrete units rather than learning subword vocabularies, has also reached speech (Zhang et al. 2023) and robot actions (Pertsch et al. 2025).
Across languages, tokenization creates measurable inequality. One study used FLORES-200, a parallel evaluation set, so the content stays the same across languages. Some languages needed up to five times more tokens than others for that content. Over-fragmented languages also scored worse in in-context learning (Ahia et al. 2023). Related work also finds unfairness between languages, and the size of the gap depends on the tokenizer and the languages compared (Petrov et al. 2023).
Monolingual tokenizers with lower fertility gave consistent gains for Arabic, Finnish, Indonesian, Korean and Turkish. Lower fertility tracked downstream gains (Spearman ρ ≈ 0.36) about as closely as pretraining corpus size did (ρ ≈ 0.38) (Rust et al. 2021).
Open questions
What makes a tokenizer good is still unsettled. Compression is the usual explanation. However, one study trained 64 English models with varied tokenizers, including PathPiece, which minimizes token count. Fewer tokens showed no clear link to better downstream accuracy (Schmidt et al. 2024). Fertility and parity (the token count for parallel text relative to English) are, at best, weak predictors of model quality (Ali et al. 2024).
BPE has also been called suboptimal for pretraining. Models using unigram LM segmentation matched or outperformed BPE-based ones on downstream tasks (Bostrom & Durrett 2020). Number handling is one concrete weakness. Commas can force right-to-left digit grouping. Doing so raised few-shot addition accuracy for the March 2023 GPT-4 checkpoint from 84.4% to 98.9% (Singh & Strouse 2024).
A related question is whether morpheme-aware segmentation should replace purely statistical splits. For Arabic classification, no tokenizer won everywhere, and even linguistically meaningless splits did surprisingly well (Alyafeai et al. 2021). Linguistic pipelines such as UDPipe take a different route. They split text into words and then assign morphological analyses in a separate trainable step, rather than learning statistical subword pieces (Straka et al. 2016).
Finally, tokenizer-free models try to remove the fixed vocabulary altogether. Canine works directly from characters (Clark et al. 2022). Charformer learns soft segmentation from bytes. It roughly matches subword models and runs 10–93% faster than a byte-level baseline (Tay et al. 2021). Whether such models can handle non-concatenative morphology, and stay efficient at scale, remains open.
Key papers
Related concepts
- Language model — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- Minimum description length — Minimum description length (MDL) is a model-selection principle rooted in information theory. It says the best explanation of a dataset is…
- Zipf's law — Zipf's law is the empirical regularity that, when the items of a collection such as the words of a text are ranked by frequency, each…
- Transformer and attention — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
More papers
Topics: ✂️ Tokenization & segmentation · 🧩 Morphology · 🧠 Language models & pretraining
Shelves: 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/subword-tokenization/