CharacTer: Translation Edit Rate on Character Level
What it is about
CharacTER is a translation edit rate computed on characters. It keeps the word-level phrase shift of TER, so a moved phrase costs something, but the edit distance after the shifts is a character-level Levenshtein distance, with spaces counted as characters. Shifting characters directly was about 44 times slower, so the shift is done on words. Three choices make it work better than TER on morphologically rich languages: the cost of a shift is the average word length of the moved phrase, two words match when identical or within one edit (for stem variants such as code and codes), and the error count is normalised by the hypothesis length instead of the reference length. The last change removes the bias of TER towards short outputs and was the largest single gain, up to about nine points in correlation. On WMT13 and WMT14 system-level correlations, the variant beats chrF and BEER on several target languages, by up to about seven points, while costing roughly ten per cent more time than TER.
Ideas
- I1. A word-shift and character-edit score for the translator's normal mode, so that hath against has or thee against thou costs a small edit and not a full miss. Compare with exact match on held-out verses, and run twice.
- I2. Test both normalisers on the Bible pairs. The hypothesis length rewards longer outputs, so adopt it only if it still wins once that bias is checked.
- I3. Charge a moved phrase by its average word length, so that displaced content words cost more than displaced function words. This is a count-based way to score the word-order residual class.
- I4. Use the one-edit stem rule as a near-match flag in the lexicon report, so that a plural or suffix variant is shown as a near merge and not as a failure.
For Mova
The places are tools/translate (measure.rs) for the normal-mode score and en/src/morph.rs for suffix tolerance. It touches open problem 5 (number and person forms that a merge such as thee/thou/ye loses). The paper is about MT scoring, not about morphology, so only the shift cost, the normaliser and the stem tolerance are taken over.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2342.pdf
Summary
Automatic machine translation evaluation still relies heavily on BLEU and TER, while character-based metrics such as CHRF have recently matched human judgements more closely. This paper introduces CharacTER, a character-level variant of TER. Running TER's shift search directly on characters is slow and harmful, because characters match too easily and shifts break words into fragments. The authors therefore keep phrase shifts at the word level, then split the shifted hypothesis into characters (spaces included) and compute Levenshtein distance. A shift costs the mean word length of the moved phrase. Words within edit distance 1 can count as matches, which catches shared stems. The total is divided by hypothesis length rather than reference length. The authors measured system-level correlation with human rankings on the WMT13–WMT15 metric tasks and the WMT16 metrics task. CharacTER is strongest when translating from English into morphologically rich languages, beating CHRF by up to 7%. It does not lead on most into-English averages. Normalizing by hypothesis length also improves plain TER, a simple change with practical value.
Key points
- Efficiency: On the paper's example, naive character-level shifting (CER) took 44 times as long as TER. The word-level shift heuristic adds only about 10% compute.
- Normalization: Dividing by hypothesis length instead of reference length raised human correlation for both TER and CharacTER by up to 9%.
- WMT14 English→German: CharacTER reached a Pearson correlation of 0.757. CHRF3 reached 0.425, and BLEU, TER, NIST and METEOR all scored ≤0.324.
- Averages by direction:
- Out of English: CharacTER averaged 0.934 on WMT13 (CHRF3: 0.861) and 0.941 on WMT15 (CHRF3 and BEER: 0.916).
- Into English: CharacTER trailed, scoring 0.939 on WMT15 versus 0.969 for BEER and METEOR.
- WMT16 metrics task: CharacTER was the top metric on half of the language pairs. It was especially strong for English↔German (en-de 0.915, de-en 0.985) and English↔Romanian (en-ro 0.959, ro-en 0.970).
Cite
@inproceedings{wang2016character,
title = {{CharacTer: Translation Edit Rate on Character Level}},
author = {Wang, Weiyue and Peter, Jan-Thorsten and Rosendahl, Hendrik and Ney, Hermann},
booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
year = {2016},
doi = {10.18653/v1/w16-2342},
url = {https://doi.org/10.18653/v1/w16-2342}
}References
- Miloš Stanojević and Khalil Sima’an. 2014. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore, MA, USA, June. find
- Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Statistical Machine Translation, Berlin, Germany, August. find our card
- Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar, 2015. Results of the WMT15 Metrics Shared Task. Association for Computational Linguistics. find
- George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 128–132, San Diego, CA, USA, March. find
- Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the ACL 2007 Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic, June. find our card
- Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. find
- Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MA, USA, June. find our card
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA, July. find our card
- Karl Pearson. 1895. Notes on Regression and Inheritance in the Case of Two Parents. In Proceedings of the Royal Society of London, volume 58, pages 240–242, London, UK, June. find
- Maja Popović. 2015. CHRF: Character n-gram F-Score for Automatic MT Evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisboa, Portugal, September. find our card
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study find