Elamur Research⌕
☰
home / papers / 2005 / On Some Pitfalls in Automatic Evaluation and Significance Testing for MT

On Some Pitfalls in Automatic Evaluation and Significance Testing for MT

Summary

The authors examine how to reliably detect very small differences between statistical machine translation (SMT) systems. They also test whether statistical significance tests assess such differences correctly. For the experiments, a phrase-based system was built on the Europarl corpus. A discriminative reranker operates on top of its 1000-best translation lists: a log-linear model with ℓ1 regularization and incremental feature selection. Different variants of the reranker were compared using three metrics: BLEU, NIST and dependency-based F-score. In addition, the authors compared the bootstrap test with approximate randomization. The results are as follows: only NIST detected a statistically significant difference between the systems. Bootstrap yields underestimated p-values and therefore more often falsely finds a difference where there is none (a Type I error). Separately, the authors highlight the problem of multiple pairwise comparisons. Their advice: for fine-grained comparisons, combine NIST with approximate randomization and use a stricter significance threshold than is customary. The work is important because it calls into question the widespread practice of presenting BLEU gains of tenths of a percentage point as system improvements.

Key points

  • Data and system. The Europarl corpus. For speed, only sentences 5–15 words long were used: 160,000 in the training set and 2,000 in the development set. The phrase table contains 200,000 phrases of up to 3 words. The reranker uses 8 baseline features from PHARAOH and a large number of overlapping phrasal features.
  • Metric comparison. After reranking, NIST rose from 6.43 to 6.58 (p < .0001). BLEU, by contrast, dropped slightly (.301 → .298, p ≈ .1–.16), as did F-score (.385 → .383, p ≈ .42). Both of the latter shifts are statistically insignificant.
  • Type I error. The authors created practically equivalent system variants that differ only in redundant features. On these, bootstrap gave p < .015 in 3 of 5 comparisons, i.e., it falsely found a difference. Approximate randomization gave p ≥ .025 in all cases.
  • Multiple comparisons. Take 15 systems: that is 105 pairwise comparisons. If each is tested at the .05 level, the probability of at least one false "significant" result reaches ≈ .9954. For 5 comparisons, a threshold of .015 keeps the overall error at ≈ .07. If the threshold has to be lowered too drastically, the authors recommend the post-hoc tests of Scheffé or Tukey.
  • Recommendation. NIST favors sensitivity over expressiveness: it takes word choice into account, but not word order or sentence structure. Therefore, for small differences it should be combined with approximate randomization and stricter significance thresholds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · AI & machine learning

Topics Machine translation Evaluation & benchmarks

Cite

Cite as Stefan Riezler and John T. Maxwell. 2005. On Some Pitfalls in Automatic Evaluation and Significance Testing for MT. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{riezler2005some,
  title = {{On Some Pitfalls in Automatic Evaluation and Significance Testing for MT}},
  author = {Stefan Riezler and John T. Maxwell},
  booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
  year = {2005},
  url = {https://aclanthology.org/W05-0908/}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nancy Chinchor, Lynette Hirschman, and David D. Lewis. 1993. Evaluating message understanding systems: An analysis of the third message understanding conference (MUC-3). Computational Linguistics, 19(3):409–449. find
  2. Paul R. Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, MA. find
  3. Richard Crouch, Ronald M. Kaplan, Tracy H. King, and Stefan Riezler. 2002. A comparison of evaluation metrics for a broad-coverage stochastic parser. In Proceedings of the ”Beyond PARSEVAL” Workshop at the 3rd International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Spain. find
  4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the ARPA Workshop on Human Language Technology. find
  5. Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman and Hall, New York. find
  6. Kevin Knight and Yaser Al-Onaizan. 1999. A primer on finite-state software for natural language processing. Technical report, USC Information Sciences Institute, Marina del Rey, CA. find
  7. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda. find
  8. Philipp Koehn. 2002. Europarl: A multilingual corpus for evaluation of machine translation. Technical report, USC Information Sciences Institute, Marina del Rey, CA. find
  9. Philipp Koehn. 2004a. PHARAOH. a beam search decoder for phrase-based statistical machine translation models. user manual. Technical report, USC Information Sciences Institute, Marina del Rey, CA. find
  10. Philipp Koehn. 2004b. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP’04), Barcelona, Spain. our card
  11. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA. our card
  12. Eric W. Noreen. 1989. Computer Intensive Methods for Testing Hypotheses. An Introduction. Wiley, New York. find
  13. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02), Philadelphia, PA. our card
  14. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  15. Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In Proceedings of the 1999 Conference on Empirical Methods in Natural Language Processing (EMNLP’99). our card
  16. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Ketherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA. our card
  17. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report IBM Research Division Technical Report, RC22176 (W0190-022), Yorktown Heights, N.Y. our card
  19. Stefan Riezler and Alexander Vasserman. 2004. Incremental feature selection and 1 regularization for relaxed maximum-entropy modeling. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP’04), Barcelona, Spain. find
  20. Stefan Riezler, Tracy H. King, Richard Crouch, and Annie Zaenen. 2003. Statistical sentence condensation using ambiguity packing and stochastic disambiguation methods for lexical-functional grammar. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda. find
  21. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA. find
  22. Sidney Siegel. 1988. Nonparametric Statistics for the Behavioral Sciences. Second Edition. MacGraw-Hill, Boston, MA. find
  23. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, Denver, CO. find
  24. Ying Zhang, Stephan Vogel, and Alex Waibel. 2004. Interpreting BLEU/NIST scores: How much improvement do we need to have a better system? In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riezler-2005-some-pitfalls-automatic-evaluation-significance/