Character-based Neural Machine Translation
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-2058.pdf
Summary
Neural machine translation (MT) struggles with large vocabularies and with morphologically rich languages, because each source word normally gets its own lookup embedding from a fixed list. The authors start from the attention-based encoder-decoder of Bahdanau et al. (2015), built with DL4MT. They change only the source side: each source word's vector is now computed from its characters. They borrow the architecture of Kim et al. (2016), in which character embeddings pass through 1D convolution filters of several widths, then max pooling, then two highway layers. Source words are never unknown under this scheme, and the model can pick up affix information. The target side still uses a standard word vocabulary.
On the WMT German-English task, the character-based system beats a standard neural baseline and a Moses phrase-based system in both directions. The larger gain is for English-to-German, even though English is not morphologically rich. The work shows that cheap character modelling on the source side alone can improve translation quality.
Key points
- Data: WMT German-English training data from EPPS, NEWS and Commoncrawl, about 3.5M sentence pairs. Preprocessing covered tokenization, truecasing, punctuation normalization and language-based filtering.
- Character encoder: convolution filters span 1–7 characters, followed by max pooling and two highway layers. Its output replaces the usual source word embedding.
- Neural baseline settings: embedding size 620, hidden size 1024, batch size 32, no dropout, 90k-word vocabulary. Unknown target words (UNKs) are replaced via source alignment ("+Src").
- BLEU, De→En:
- Phrase-based: 20.99
- NN: 18.83
- NN+Src: 20.64
- CHAR: 21.40
- CHAR+Src: 22.10
- BLEU, En→De:
- Phrase-based: 17.04
- NN: 16.47
- NN+Src: 17.15
- CHAR: 19.53
- CHAR+Src: 20.22
- Unknown target words after post-processing (NN → CHAR):
- De→En: 1491 → 1260
- En→De: 3148 → 2640
- Qualitative examples: the gains appear as better adequacy, handling of morphological variants, and word-sense disambiguation.
Cited by
- Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging
- Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Findings of the 2019 Conference on Machine Translation (WMT19)
- What do Neural Machine Translation Models Learn about Morphology?
- Spelling Normalization of Historical Documents by Using a Machine Translation Approach
- Sequence-Level Knowledge Distillation
- Fully Character-Level Neural Machine Translation without Explicit Segmentation
- Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
- DyNet: The Dynamic Neural Network Toolkit
Cite
@inproceedings{costajussa2016character,
title = {{Character-based Neural Machine Translation}},
author = {Costa-jussà, Marta R. and Fonollosa, José A. R.},
booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
year = {2016},
doi = {10.18653/v1/p16-2058},
url = {https://doi.org/10.18653/v1/p16-2058}
}