Neural Machine Translation with Byte-Level Subwords

Summary

Most NMT vocabularies are built from characters. Languages with very large character inventories, such as Japanese and Chinese, and noisy text therefore spend many vocabulary slots on rare symbols. Feeding raw bytes (256 symbols) avoids this but makes sequences too long to be practical. Wang, Cho and Gu (Facebook AI Research) propose byte-level BPE (BBPE). They learn BPE merges over UTF-8 bytes with SentencePiece, so rare characters break into byte fragments and no token is ever unknown. Because these fragments can be partial characters, each embedding is first contextualized by a depth-wise convolution or a Bi-GRU before entering a Fairseq Transformer. A dynamic-programming routine also salvages valid characters from malformed byte output. The authors test on WMT 2014 En-De, Ja-En (KFTT, TED, JESC), Si-En from FLoRes and a 58-source-language TED Talks X-En set. BBPE reaches BPE-level quality with about one-eighth of the vocabulary. It improves multilingual translation by sharing symbols across scripts and allows transfer to a language with an unseen script. This makes it relevant for compact, script-agnostic and low-resource systems.

Key points

Data: En-De uses WMT 2014 (4.5M training sentences). Ja-En has 3.5M sentences, Si-En has 405K, and X-En has 5.1M training sentences with a 165K-sentence test set.
Contextualization: On X-En validation, adding a Bi-GRU over BBPE embeddings yields a relative BLEU gain of more than 4% for every BBPE size. The authors make this the default setting, written as "BBPE <size>+".
Splitting rare characters: A large share of tokens are partial characters on languages with big character sets. With a 4K vocabulary this is 47.6% on Ja-En and 39.1% on X-En, versus 4.9% on En-De. Even very small BBPE vocabularies produce much shorter sequences than character-level tokenization.
Ja-En results (T base, combined test set): BBPE 4K+ scores 16.80 BLEU, close to BPE 16K+ at 17.19 and above Char+ at 16.43 and Byte+ at 16.27.
Transfer learning: A BBPE 4K+ model pretrained on X-En and fine-tuned on Si-En (Sinhala script unseen during pretraining) improves on Si-En-only baselines (about 7.1–7.2 BLEU) by 0.9–1.8 BLEU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics Machine translation Tokenization & segmentation

Institutions

🏢MetaMenlo Park ·

United States137 works · 196,530 citations

Cite

Cite as Changhan Wang, Kyunghyun Cho and Jiatao Gu. 2019. Neural Machine Translation with Byte-Level Subwords. arXiv:1909.03341.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.03341 · arXiv 1909.03341

@misc{wang2019neural,
  title = {{Neural Machine Translation with Byte-Level Subwords}},
  author = {Changhan Wang, Kyunghyun Cho and Jiatao Gu},
  year = {2019},
  doi = {10.48550/arxiv.1909.03341},
  eprint = {1909.03341},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.03341}
}

References

6 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. our card
Cettolo, M.; Girardi, C.; and Federico, M. 2012. Wit3: Web inventory of transcribed and translated talks. In Conference of European Association for Machine Translation, 261–268. our card
Chen, M. X.; Firat, O.; Bapna, A.; Johnson, M.; Macherey, W.; Foster, G.; Jones, L.; Parmar, N.; Schuster, M.; Chen, Z.; et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. arXiv preprint arXiv:1804.09849. our card
Chenguang Wang, Mu Li, A. J. S. 2019. Language models with transformers. In ArXiv e-prints. find
Cherry, C.; Foster, G.; Bapna, A.; Firat, O.; and Macherey, W. 2018. Revisiting character-based neural machine translation with capacity and compression. arXiv preprint arXiv:1808.09943. find
Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Con-Ask̂questions,̂demand̂proof,̂demand̂evidence. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2019-neural-machine-translation-byte-level/

📄 Neural Machine Translation with Byte-Level Subwords