Elamur Research
☰
home / papers / 2004 / Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics

Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P04-1077.pdf

Summary

BLEU and NIST, the standard automatic MT metrics, score translations by fixed-length consecutive n-gram overlap. This has drawbacks. It ignores sentence-level word order beyond local windows and needs a preset maximum n. It uses a brevity penalty in place of genuine recall, and it often gives individual sentences a score of zero. Lin and Och propose order-aware alternatives that compare a candidate translation with one or more references:

  • ROUGE-L builds an F-measure from the longest common subsequence. It rewards words that appear in the same order even when gaps separate them.
  • ROUGE-W weights that subsequence so that consecutive runs count more.
  • ROUGE-S counts shared skip-bigrams, meaning ordered word pairs at any distance or within a limited one.

When several references are available, the best-matching one is used. The authors compared these metrics with BLEU, NIST, GTM, PER and WER by correlating them with human adequacy and fluency judgments. Bootstrap confidence intervals were computed over 919 sentence segments. ROUGE-S* gave the strongest adequacy correlation. ROUGE-L and several ROUGE-S variants matched BLEU on fluency and clearly beat it on adequacy. They also behave sensibly on single sentences, which helps error diagnosis.

Key points

  • ROUGE-S* (no skip-distance limit) reached a Pearson correlation of 0.95 with adequacy on lowercased, stemmed text, up from 0.85 when case was kept. WER was the weakest metric at 0.48.
  • BLEU12 had the highest Pearson correlation with fluency (0.93). However, it was statistically indistinguishable from most other metrics, except WER and GTM10.
  • Stemming raised correlation with adequacy, while preserving case raised correlation with fluency. For example, ROUGE-S* scored 0.84 on fluency with case kept and 0.78 with stemming.
  • By Spearman's coefficient, ROUGE-L was the best metric for both adequacy and fluency. In a toy example where BLEU-2 ties two candidates, ROUGE-L separates them (0.75 vs 0.5).
  • Planned extensions include soft matching of synonyms and paraphrases, for example via WordNet. The same metrics were also applied to summarization evaluation (Lin 2004).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

211 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 207 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1218955.1219032

@inproceedings{lin2004automatic,
  title = {{Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics}},
  author = {Lin, Chin-Yew and Och, Franz Josef},
  booktitle = {Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)},
  year = {2004},
  doi = {10.3115/1218955.1219032},
  url = {https://doi.org/10.3115/1218955.1219032}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiba, Y., K. Imamura, and E. Sumita. 2001. Using Multiple Edit Distances to Automatically Rank Machine Translation Output. In Proceedings of the MT Summit VIII, Santiago de Compostela, Spain. find
  2. Barzilay, R. and L. Lee. 2003. Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignmen. In Proceeding of NAACL-HLT 2003, Edmonton, Canada. find
  3. Leusch, G., N. Ueffing, and H. Ney. 2003. A Novel String-to-String Distance Measure with Applications to Machine Translation Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. find
  4. Levenshtein, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady. find
  5. Lin, C.Y. 2004. R OUGE : A Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out, post-conference workshop of ACL 2004, Barcelona, Spain. find
  6. Lin, C.-Y. and F. J. Och. 2004. O RANGE : a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of 20 th International Conference on Computational Linguistic (COLING 2004), Geneva, Switzerland. find
  7. Miller, G. 1990. WordNet: An Online Lexical Database. International Journal of Lexicography, 3(4). find
  8. Melamed, I.D. 1995. Automatic Evaluation and Uniform Filter Cascades for Inducing N-best Translation Lexicons. In Proceedings of the 3 rd Workshop on Very Large Corpora (WVLC3). Boston, U.S.A. find
  9. Melamed, I.D., R. Green and J. P. Turian. 2003. Precision and Recall of Machine Translation. In Proceedings of NAACL/HLT 2003, Edmonton, Canada. find
  10. Nießen S., F.J. Och, G, Leusch, H. Ney. 2000. An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation, Athens, Greece. find
  11. NIST. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics. AAAAAAAAAAA http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf ↗
  12. Pantel, P. and Lin, D. 2002. Discovering Word Senses from Text. In Proceedings of SIGKDD-02. Edmonton, Canada. find
  13. Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2001. B LEU : a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176 (W0109-022). find
  14. Porter, M.F. 1980. An Algorithm for Suffix Stripping. Program, 14, pp. 130-137. find
  15. Saggion H., D. Radev, S. Teufel, and W. Lam. 2002. Meta-Evaluation of Summaries in a Cross-Lingual Environment Using Content-Based Metrics. In Proceedings of COLING-2002, Taipei, Taiwan. find
  16. Su, K.-Y., M.-W. Wu, and J.-S. Chang. 1992. A New Quantitative Quality Measure for Machine Translation System. In Proceedings of COLING-92, Nantes, France. find
  17. Thompson, H. S. 1991. Automatic Evaluation of Translation Quality: Outline of Methodology and Report on Pilot Experiment. In Proceedings of the Evaluator’s Forum, ISSCO, Geneva, Switzerland. find
  18. Turian, J. P., L. Shen, and I. D. Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. find
  19. Van Rijsbergen, C.J. 1979. Information Retrieval. Butterworths. London. find