SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-2012.pdf

Summary

Neural text systems such as NMT still often depend on preprocessing that is tied to specific languages. Examples include the rule-based Moses tokenizer and the separate word segmenters needed for Chinese, Japanese and Korean, which make multilingual and reproducible pipelines harder to build. Kudo and Richardson present SentencePiece, an open-source subword tokenizer and detokenizer written in C++ with Python and TensorFlow bindings. It learns BPE or unigram language model vocabularies directly from raw sentences, so no pre-tokenization is needed. Whitespace is treated as an ordinary symbol (U+2581), which makes tokenization fully reversible. The tool also handles vocabulary-to-id mapping and NFKC-based normalization, which users can customize. Everything, including the normalization rules, is stored in a single self-contained model file. In English-Japanese experiments on KFTT with GNMT, subword models beat word-level baselines. Training on raw text matched or beat pre-tokenized training, and the gain was largest when Japanese was the target language. This matters because a single tool can now preprocess any language, including unsegmented scripts and morphologically rich languages such as Ukrainian, with no hand-written rules.

Key points

• Supports two segmentation algorithms (BPE and unigram LM). BPE segmentation uses a binary heap, which brings its cost down from O(N²) to O(N log N). Users set the final vocabulary size (--vocab_size) rather than a number of merge operations.
• Lossless design: decoding an encoded normalized string returns exactly that normalized string. Detokenization only needs to join the pieces and turn the meta symbol back into spaces.
• On KFTT (440k training sentences), shared 8k SentencePiece vocabularies reached 29.55 BLEU for ja→en and 21.62 for en→ja. The 80k word baselines scored 28.24 and 20.06. Adding pre-tokenization lowered en→ja to 20.86.
• On raw Japanese text, SentencePiece trained in 217.3 s versus 528.0 s for subword-nmt. It segmented in 5.9 s versus 216.2 s.
• Released under Apache 2 at github.com/google/sentencepiece. Models are serialized as Protocol buffers, and the API supports on-the-fly subword sampling for subword regularization.