Elamur Research⌕
☰
home / concepts / Tokenization

🪙 Tokenization

Concept · 30 key papers · all concepts

Part of: 🗨️ Natural language processing

Tokenization is the step in text processing that splits a stream of characters into units called tokens, such as words, punctuation marks, numbers or subword pieces. These tokens become the basic items that later analysis or a language model counts, tags and predicts.

Also: word segmentation, lexical analysis, pre-tokenization, subword segmentation, token segmentation

What it is

Almost every program that reads text first has to decide where one unit ends and the next begins. In natural language processing this step is called tokenization. The sentence "Dr. Smith's car didn't start." looks simple. A tokenizer still has to decide several things:

The Penn Treebank convention, for instance, splits don't into do + n't (Mielke et al. 2021).

The difficult cases fall into a few groups:

A related task is sentence segmentation: deciding which periods, question marks or line breaks end a sentence. It is often solved together with tokenization (Straka et al. 2016).

Tokens should be kept apart from types. A token is one occurrence in running text, and a type is a distinct form. "the cat saw the dog" has five tokens but four types. The ratio between the two is the type–token ratio, and how often types recur is described by Zipf's law.

The key idea

At bottom, a tokenizer makes a binary decision at every gap between characters: boundary or no boundary. Older systems made that decision with hand-written rules, such as regular expressions, abbreviation lists and exceptions for clitics. The Moses tokenizer used in machine translation is the classic example (Kudo & Richardson 2018).

Learned systems turn the decision into classification. UDPipe gives each character one of three labels: no boundary, token boundary, or sentence boundary. It learns these labels with a bidirectional GRU, trained only by aligning raw text with its tokenized version (Straka et al. 2016). In Chinese word segmentation the same idea appears as position tags. Each character is marked as word-initial, word-internal, word-final, or a single-character word (Xue 2003).

A second key idea is that the "right" segmentation depends on its purpose. For language models the goal is a vocabulary that is small but open, so every string can be represented. A standard way to measure how much a tokenizer splits words is fertility:

fertility = T / W

Here T is the number of tokens the tokenizer produces for a text, and W is the number of whitespace-delimited words in the same text. A value near 1 means words stay whole, and higher values mean heavier fragmentation. Lower fertility in monolingual tokenizers went along with better downstream results in several languages (Rust et al. 2021).

Try it: Mova’s tokens against GPT-2’s

Type a sentence and see how two tokenizers cut it. Mova’s English tokenizer follows the Universal Dependencies conventions: words, punctuation and clitics (did + n’t), abbreviations kept whole (U.S.). GPT-2’s byte-pair encoding cuts by frequency: common words stay whole, a leading space is part of the token, rare words fall into pieces. Both run in your browser; the text is not sent anywhere.

Six of the examples, token by token (Mova’s tokenizer, then GPT-2’s BPE; ␣ marks a space that belongs to the token):

Mr. Smith didn't go to the U.S. today.
Mova, 10 tokens: Mr. Smith did n't go to the U.S. today .
GPT-2 BPE, 14 tokens: Mr . ␣Smith ␣didn 't ␣go ␣to ␣the ␣U . S . ␣today .
I cannot believe it's already 10:30 — we're late!
Mova, 13 tokens: I can not believe it 's already 10:30 — we 're late !
GPT-2 BPE, 14 tokens: I ␣cannot ␣believe ␣it 's ␣already ␣10 : 30 ␣— ␣we 're ␣late !
The state-of-the-art model costs $3.50 per 1,000 tokens, i.e. 10% more...
Mova, 21 tokens: The state - of - the - art model costs $ 3.50 per 1,000 tokens , i.e. 10 % more ...
GPT-2 BPE, 30 tokens: The ␣state - of - the - art ␣model ␣costs ␣$ 3 . 50 ␣per ␣1 , 000 ␣to k ens , ␣i . e . ␣10 % ␣more ...
Tokenizers split "unbelievably" into pieces; BPE doesn't know morphology.
Mova, 14 tokens: Tokenizers split " unbelievably " into pieces ; BPE does n't know morphology .
GPT-2 BPE, 22 tokens: T oken izers ␣split ␣" un bel iev ably " ␣into ␣pieces ; ␣B PE ␣doesn 't ␣know ␣mor ph ology .
E-mail a@b.com or visit https://research.elamur.ai/free/en/ for the 1990s data.
Mova, 11 tokens: E-mail a@b.com or visit https://research.elamur.ai/free/en / for the 1990s data .
GPT-2 BPE, 32 tokens: E - mail ␣a @ b . com ␣or ␣visit ␣https :// re se arch . el am ur . ai / free / en / ␣for ␣the ␣1990 s ␣data .
She'll say: "John's dog won't bark at the 2nd door."
Mova, 17 tokens: She 'll say : " John 's dog wo n't bark at the 2nd door . "
GPT-2 BPE, 18 tokens: She 'll ␣say : ␣" John 's ␣dog ␣won 't ␣b ark ␣at ␣the ␣2 nd ␣door ."

Mova’s tokenizer: a table-driven automaton with dictionaries learned from the English Web Treebank and the ESL spoken treebank of Universal Dependencies (CC BY-SA 4.0). GPT-2: OpenAI’s vocabulary (© 2019 OpenAI, Modified MIT License), its first 12,000 merges of 50,000 — a word the full vocabulary keeps whole can split here.

History and founding papers

Rule-based tokenizers and the word problem

Early English pipelines treated tokenization as nearly solved: split on whitespace, then peel off punctuation by rule. Languages written without spaces showed how much was hidden in that assumption. Sproat, Shih, Gale and Chang cast Chinese segmentation as a search over weighted finite-state transducers. Their model combined a dictionary with sub-models for derived words, personal names and transliterated foreign names, and was built for a Mandarin text-to-speech system (Sproat et al. 1996). Their work also showed that human annotators disagree about where Chinese words end (Sproat et al. 1996).

Shared benchmarks and character tagging

The SIGHAN bakeoffs made Chinese segmentation comparable across groups. The first, in 2003, offered four corpora, each with its own segmentation standard, plus closed and open tracks (Sproat & Emerson 2003). Further rounds followed in 2005 (Emerson 2005) and 2006, the latter adding named entity recognition (Levow 2006). Across the rounds, out-of-vocabulary words stayed the main weakness (Emerson 2005).

Xue's reformulation of segmentation as character tagging with a maximum entropy model became the dominant approach (Xue 2003). Zhang and Clark then trained segmentation and part-of-speech tagging jointly with a single perceptron, which reduced the errors that a pipeline passes from one step to the next (Zhang & Clark 2008). Chang, Galley and Manning found that a higher segmentation F-score does not reliably give better translations. Consistency and granularity of the units mattered more (Chang et al. 2008).

For Arabic, Habash and Rambow handled tokenization, tagging and morphological disambiguation in one step, choosing among the outputs of a morphological analyzer (Habash & Rambow 2005).

Unsupervised and neural segmentation

Bayesian models learned word boundaries from unsegmented text or phonemic transcripts. Goldwater, Griffiths and Johnson showed that bigram context matters once search is done properly (Goldwater et al. 2006). Mochihashi, Yamada and Ueda nested a character-level Pitman-Yor model inside a word-level one (Mochihashi et al. 2009). Neural taggers replaced hand-built feature templates with character embeddings (Zheng et al. 2013), and later with LSTM memory (Chen et al. 2015).

Segmenting above the word

Segmentation also happens at larger scales. Choi's C99 split documents into topical blocks using ranked sentence similarities (Choi 2000). Utiyama and Isahara cast the same task as a probabilistic search with a minimum description length prior (Utiyama & Isahara 2001). Pevzner and Hearst diagnosed biases in the Pk metric used for such tasks and proposed WindowDiff (Pevzner & Hearst 2002). Koshorek and colleagues made topic segmentation supervised by using Wikipedia tables of contents as labels (Koshorek et al. 2018).

In speech translation, the system's sentence boundaries rarely match the reference. Matusov and colleagues re-segmented MT output so that standard metrics such as BLEU could still be applied (Matusov et al. 2005).

The step to subwords

Neural machine translation exposed the cost of closed word vocabularies. Sennrich, Haddow and Birch adapted byte pair encoding to split rare words into subword units, so the model could translate an open vocabulary (Sennrich et al. 2016). Kudo and Richardson's SentencePiece learned such vocabularies directly from raw text. It treats whitespace as an ordinary symbol, which makes the process reversible and removes the need for language-specific pre-tokenizers (Kudo & Richardson 2018). BPEmb then released pretrained subword embeddings for 275 languages (Heinzerling & Strube 2018). The algorithms themselves are covered under subword tokenization.

How it is used today

Treebank pipelines and multiword tokens

In Universal Dependencies, tokenization is the first layer of annotation. UD separates the orthographic token from the syntactic words it contains. Spanish "del" or German "zum" is a multiword token that is split into two words, and each word gets its own lemma and node in the tree. The mapping between tokens and word forms is therefore not one-to-one (Mielke et al. 2021).

UDPipe 1.2 finds sentence and token boundaries jointly with a character-level network. It splits multiword tokens with a dictionary and suffix rules extracted from the treebank, then passes the result to part-of-speech tagging and dependency parsing (Straka & Straková 2017). It was the baseline system of the CoNLL 2017 shared task on parsing from raw text (Straka & Straková 2017).

Language models

Every large language model built on the Transformer reads subword ids, so its vocabulary shapes its cost and its coverage. Varying the tokenizer while keeping all else fixed changed both downstream accuracy and training compute. An English-centric tokenizer used for multilingual training cost up to 68% more compute (Ali et al. 2024).

The same sentence can need very different numbers of tokens in different languages. For one widely used commercial tokenizer, Bulgarian text took about 2.6 times as many tokens as its English translation (Petrov et al. 2023). Content-matched comparisons across 22 languages found gaps of up to roughly five times. The gaps were linked to script and encoding as well as to the amount of training data (Ahia et al. 2023). Because commercial APIs charge per token, such gaps translate into price differences. This concern is central to multilingual NLP and to low-resource languages.

Tokenizer-free models and beyond text

Some encoders skip the tokenizer altogether. CANINE reads Unicode characters through hash embeddings and downsampling (Clark et al. 2022). Charformer learns soft subword blocks over bytes with a gradient-based module (Tay et al. 2021).

The notion of tokenization has also spread to other kinds of data. FAST compresses chunks of robot actions with a discrete cosine transform before discretizing them, which makes high-frequency control learnable by next-token prediction (Pertsch et al. 2025).

Open questions

Papers on these problems are collected under tokenization and segmentation.

Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

More papers

Topics: ✂️ Tokenization & segmentation · 🧩 Morphology · 🧠 Language models & pretraining

Shelves: 🤖 Artificial Intelligence

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/concepts/tokenization/