Elamur Research⌕
☰
home / papers / 2004 / Minimum Bayes-Risk Decoding for Statistical Machine Translation

Minimum Bayes-Risk Decoding for Statistical Machine Translation

Summary

Statistical MT systems usually pick the most probable translation (MAP decoding), even though they are judged by metrics such as BLEU, WER or PER that capture different aspects of quality. Kumar and Byrne adapt Minimum Bayes-Risk (MBR) decoding, previously used in speech recognition and bitext word alignment, to machine translation. Instead of the single most likely output, the decoder selects the hypothesis with the lowest expected loss under a chosen loss function, averaged over the system's candidate translations. The authors organize loss functions into three tiers: lexical, target-language parse-tree, and bilingual parse-tree. They introduce a Bitree loss that compares English parse trees through their alignment to the parse of the Chinese source sentence. On a Chinese-to-English task (NIST 2001+2002 test set), each MBR decoder tends to improve the metric it was tuned for compared with the MAP baseline. This shows that a single trained model can be steered toward different evaluation criteria, including syntactic ones, without retraining.

Key points

  • The hierarchy of loss functions has three levels:

    • lexical losses use only word strings (BLEU-derived loss, WER, PER);
    • target parse-tree losses compare only the two English trees (e.g., tree-edit distance, tree kernels), and these were not tested experimentally;
    • bilingual parse-tree losses use strings, alignments and parse trees in both languages.

  • Bitree loss:

    • Each source-tree node is mapped to a target-tree node. The mapping follows the word alignment to the leftmost and rightmost aligned target words and takes their closest common ancestor.
    • Corresponding subtrees are compared with a 0/1 distance.
    • The Bitree Error Rate is this loss divided by the number of commonly aligned nodes.

  • Results (NIST 2001+2002), MAP baseline vs. MBR decoders:

    DecoderBLEUmWERmPERmBiTree
    MAP baseline31.264.941.369.0
    MBR-BLEU31.565.141.168.9
    MBR-WER31.364.340.868.5
    MBR-PER31.364.640.468.6
    MBR-BiTree30.764.141.168.0

    Confidence intervals are reported at the 70% level.

  • The WER and PER decoders beat MAP on all metrics. The BiTree decoder helps WER but hurts BLEU. The BLEU decoder matches or underperforms MAP on every metric other than BLEU.

  • MBR is applied as rescoring over N-best lists. Extending it to translation lattices is left as future work.

Interesting ideas

  • I1. Decode for the metric you care about. Turning an evaluation metric into a loss and choosing the minimum-expected-loss candidate lets one fixed model serve several objectives. This applies to any generation task with a known scorer, such as Ukrainian NLP pipelines evaluated with LAS or chrF.
  • I2. Inject linguistic knowledge through the loss, not the model. Syntax enters only through the loss function, so there is no need to build detailed syntactic models or retrain. This is cheap when you have a good parser but no budget to retrain the generator.
  • I3. Use the source parse as a pivot. Projecting both target trees onto the source tree via word alignment avoids costly unconstrained tree-to-tree alignment. The same trick could compare outputs structurally in cross-lingual tasks or treebank projection.
  • I4. Check cross-metric side effects. Evaluating every decoder under every metric reveals trade-offs, such as the BiTree decoder losing BLEU. This full evaluation matrix is a good habit whenever a system is tuned toward one criterion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

261 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 245 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kumar2004minimum,
  title = {{Minimum Bayes-Risk Decoding for Statistical Machine Translation}},
  author = {Kumar, Shankar and Byrne, William},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1022/}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AMTA. 2003. Workshop on Machine Translation Evaluation, MT Summit IX. www.issco.unige.ch/projects/isle/MTE-at-MTS9.html . find
  2. S. Bangalore, V. Murdock, and G. Riccardi. 2002. Bootstrapping bilingual data using consensus translation for a multilingual instant messaging system. In Proceedings of COLING, Taipei, Taiwan. find
  3. P. J. Bickel and K. A. Doksum. 1977. Mathematical Statistics: Basic Ideas and Selected topics. Holden-Day Inc., Oakland, CA, USA. find
  4. D. Bikel and D. Chiang. 2000. Two statistical parsing models applied to the chinese treebank. In Proceedings of the Second Chinese Language Processing Workshop, pages 1–6, Hong Kong. find
  5. P. F. Brown, J. Cocke, S. A. Della Pietra, V. J. Della Pietra, F. Jelinek, J. D. Lafferty, R. L. Mercer, and P. S. Roossin. 1990. A statistical approach to machine translation. Computational Linguistics, 16(2):79–85. find
  6. M. Collins and N. Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the weighted perceptron. In Proceedings of EMNLP, Philadelphia, PA, USA. find
  7. M. J. Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. find
  8. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. of HLT 2002, San Diego, CA. USA. find
  9. Performance Metrics mWER(%) mPER (%) mBiTree Error Rate(%) +/-0.9 +/-0.6 +/-1.0 64.9 41.3 69.0 65.1 64.3 64.6 64.1 41.1 40.8 40.4 41.1 68.9 68.5 68.6 68.0 find
  10. G. Foster, P. Langlais, and G. Lapalme. 2002. User-friendly text prediction for translators. In Proc. of EMNLP, Philadelphia, PA, USA. find
  11. V. Goel and W. Byrne. 2000. Minimum Bayes-risk automatic speech recognition. Computer Speech and Language, 14(2):115–135. find
  12. J. Goodman. 1996. Parsing algorithms and metrics. In Proc. of ACL-1996, pages 177–183, Santa Cruz, CA, USA. find
  13. JHU. 2003. Syntax for statistical machine translation, Final report, JHU summer workshop. http://www.clsp.jhu.edu/ws2003/groups/translate/ . ↗
  14. S. Kumar and W. Byrne. 2002. Minimum Bayes-Risk alignment of bilingual texts. In Proc. of EMNLP, Philadelphia, PA, USA. find
  15. S. Kumar and W. Byrne. 2003. A weighted finite state transducer implementation of the alignment template model for statistical machine translation. In Proceedings of HLT-NAACL, Edmonton, Canada. find
  16. I. D. Melamed, R. Green, and J. P. Turian. 2003. Precision and recall of machine translation. In Proceedings of the HLT-NAACL, Edmonton, Canada. find
  17. NIST. 2003. The NIST Machine Translation Evaluations. http://www.nist.gov/speech/tests/mt/ . ↗
  18. F. Och. 2002. Statistical Machine Translation: From Single Word Models to Alignment Templates. Ph.D. thesis, RWTH Aachen, Germany. find
  19. F. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, Sapporo, Japan. find
  20. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division. find our card
  21. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. 2002. Numerical Recipes in C++. Cambridge University Press, Cambridge, UK. find
  22. M. Tang, X. Luo, and S. Roukos. 2002. Active learning for statistical natural language parsing. In Proceedings of ACL 2002, Philadelphia, PA, USA. find
  23. N. Ueffing, F. Och, and H. Ney. 2002. Generation of word graphs in statistical machine translation. In Proc. of EMNLP, pages 156–163, Philadelphia, PA, USA. find

↑ back to top

This page in Markdown →