Error Analysis of Statistical Machine Translation Output
What it is about
The paper gives a hand-built error typology for statistical MT output and applies it to the RWTH system in the first TC-STAR evaluation, for Spanish-to-English, English-to-Spanish and Chinese-to-English, on speech-derived data (clean text, verbatim transcripts and recogniser output). The top level has five classes: missing words, word order, incorrect words, unknown words and punctuation. Missing words are split into content and filler words, word order into word and phrase level, local and long range, and incorrect words into wrong lexical choice, wrong disambiguation, wrong form, extra words, style and idioms. The authors observe that automatic metrics such as WER, PER, BLEU and NIST cannot say what the main problems are. For English-to-Spanish, verb tense errors made up about 15 percent of the errors, and noun-adjective reordering was prominent. For Spanish-to-English, lexical choice and disambiguation were about 28 percent, and most missing words were filler words that did not change the meaning. The typology is the basis for a later automatic classifier.
Ideas
- I1. Use this typology as the label set for the translator's residual: missing versus extra words, local and long-range order, word and phrase order, wrong lexical choice, wrong form, idiom. Most residual classes already match, so the change is in the names.
- I2. Count missing filler words apart from missing content words in
tools/translate/src/measure.rs. This is open problem 4 in another form: function words and boundaries should be features of meta tokens, and their loss needs its own number. - I3. Treat the be-type disambiguation as a sense class. The paper's example, two Spanish verbs for one English verb, is the analogue of merging two concepts that differ in use, such as be and have in the current lexicon (open problem 2).
- I4. Report verb tense and person as a separate error class for every target language with a verbal system. This is a morphology measure that
en/src/morph.rscould supply.
For Mova
The code is the residual labels in tools/translate and the morphology side in en/src/morph.rs. The typology gives the translator a human-readable error list that matches its own classes, so that residual numbers can be read in the same terms as earlier error studies. The paper's finding that filler words account for much of the missing material supports the open problem on function words.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC 4.0 (ELRA, LREC proceedings)" does not allow redistributing the text itself. Original — aclanthology.org/L06-1244.pdf
Summary
Automatic metrics such as WER, PER, BLEU and NIST can rank MT systems, but they do not show which kinds of mistakes dominate a system's output. Vilar et al. propose a hierarchical scheme for human error annotation of MT output. It builds on the typology of Llitjós et al. (2005), and its top level has five categories (Incorrect Words, Missing Words, Unknown Words, Word Order and Punctuation), each divided into more specific subtypes. The authors apply it to RWTH's phrase-based log-linear system from the first TC-STAR evaluation. They cover three directions: English→Spanish and Spanish→English on European Parliament (EPPS) data, and Chinese→English on broadcast news. For EPPS they compare three input conditions: Final Text Edition (FTE), verbatim transcripts and ASR output. The dominant error type turns out to depend on the language pair. Verb tense is the main problem when translating into Spanish. For Chinese→English, lexical choice, missing words and reordering matter most. The paper argues that this kind of diagnosis is needed to decide where research effort should go.
Key points
- System: a log-linear combination of seven models.
- Core models: phrase models in both directions plus a target language model.
- Also included: phrase-level IBM1 models and phrase and length penalties.
- N-best lists are rescored with sentence-level IBM1 models and clustered LMs.
- EPPS data: the same FTE training corpus served all three test conditions, with only the preprocessing differing. This mismatch raises error rates on the Verbatim and ASR inputs.
- English→Spanish: wrong verb tense makes up 15.1% of all errors. Confusing "ser" and "estar" is a notable disambiguation problem.
- Chinese→English data: LDC "Large Data Condition" corpora, with over 7M training sentences (about 200M Chinese running words). Test sets come from Voice of America transcriptions, with 494 sentences each. Because of the very low OOV rate, no Unknown Words errors were found.
- Chinese→English FTE error shares:
- Incorrect Words 27.9%, of which Wrong Lexical Choice is 18.5%.
- Missing Words 27.5%.
- Word Order and Extra Words 17.8% each.
- Named entities 8.9%, of which person names are 5.4%.
- Long-range reorderings 10.6% versus 7.3% local.
- On verbatim input, Missing Words drops to 20.9%, while Incorrect Words (32.0%) and Extra Words (19.8%) rise.
Interesting ideas
- Severity split for missing words: separate omissions that destroy meaning from omissions that only break grammar. This lets a team rank fixes by impact rather than raw counts, which suits any generation task.
- Language-specific subcategories inside a shared taxonomy: keep one common taxonomy but add subcategories only for the pair being studied. Examples are tense versus agreement for Spanish, and named-entity types or sentence-type reordering for Chinese. The scheme then stays comparable across pairs while still capturing what matters for a highly inflected target such as Ukrainian.
- Two reordering dimensions: classify order errors by whether single words or whole blocks must move, and by whether the move stays within a syntactic chunk. The result points directly to whether a system needs local or phrase-level reordering models.
- Same system, three input conditions: evaluate one system on clean text, verbatim transcripts and ASR output. This isolates how spontaneous-speech effects and recognition errors shift the error profile, a useful setup for any speech-to-text pipeline.
- Reading error categories together: relate error classes to each other and to metrics. Examples include extra filler words compensating for missing ones, and the WER–PER gap reflecting reordering errors. This helps interpret automatic scores rather than treating them as opaque numbers.
Cite
@inproceedings{vilar2006error,
title = {{Error Analysis of Statistical Machine Translation Output}},
author = {Vilar, David and Xu, Jia and D'Haro, Luis Fernando and Ney, Hermann},
booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
year = {2006},
doi = {10.63317/453fcbexxngk},
url = {https://doi.org/10.63317/453fcbexxngk}
}References
- George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. ARPA Workshop on Human Language Technology. find
- LDC. 2005. Linguistic data consortium chinese training data resources. http://www.ldc.upenn.edu/Projects/TIDES/mt2005cn.htm. ↗
- Ariadna Font Llitjós, Jaime G. Carbonell, and Alon Lavie. 2005. A framework for interactive and automatic refinement of transfer-based machine translation. In Proc. of the 10th Annual Conf. of the European Association for Machine Translation (EAMT), Budapest, Hungary, May. find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, July. find our card
- David Vilar, Evgeny Matusov, Saša Hasan, Richard Zens, and Hermann Ney. 2005. Statistical Machine Translation of European Parliamentary Speeches. In Proceedings of MT Summit X, pages 259–266, Phuket, Thailand, September. Asia-Pacific Association for Machine Translation (AAMT). find