Elamur Research⌕
☰
home / papers / 2007 / Combining Outputs from Multiple Machine Translation Systems

Combining Outputs from Multiple Machine Translation Systems

Paper · 2007 · Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang et al.

Summary

The paper asks whether translations from several machine translation systems built on different paradigms can be merged into output that beats any single one of them. The authors combined six Arabic- and Chinese-to-English systems: three phrasal, two hierarchical and one syntax-based. They tested three combination strategies.

  • Sentence level: re-rank a pooled N-best list using confidence scores and a 5-gram LM.
  • Phrase level: build a per-sentence phrase table from the systems' source alignments and re-decode it with a phrasal decoder.
  • Word level: apply consensus network decoding, using TER-based alignment and a skeleton chosen by minimum Bayes risk.

All weights are tuned with Powell's method, targeting TER or BLEU. Evaluation used NIST MT05 and the newsgroup part of the GALE 2006 dry-run. Sentence- and phrase-level combination gave small gains that mostly did not carry over to the test sets. Word-level combination was the most dependable and helped even with no tuned weights. The best results came from feeding the sentence- and phrase-level outputs into the word-level step as extra inputs. This matters because systems of similar accuracy make different errors, and those differences can be exploited.

Key points

  • Six systems were combined: three phrasal, two hierarchical and one syntax-based. Testing was done on NIST MT05 (newswire) together with the newsgroup data from the GALE 2006 dry-run.
  • Tuning data and metrics:
    • Sentence- and phrase-level weights were tuned on NIST MT03.
    • Six-way word-level weights were tuned on MT03 plus MT04, and eight-way weights on MT04 only.
    • Arabic was tuned for TER and Chinese for BLEU, because optimizing TER on Chinese clearly lowered BLEU.
  • On the tuning sets, word-level combination improved TER by about 2–3% and BLEU by about 2–4%. The other two methods improved both metrics by only about 0.5–1%.
  • Chinese newswire results:
    • The 8-way BLEU-tuned combination ("BLEU 8") reached about 36.9 BLEU, against 35.3 for the strongest single system.
    • Unweighted 6-way consensus decoding gave the lowest TER, about 53.8.
  • In internal HTER checks, BLEU 8 needed fewer edits after human post-editing than the unweighted 6-way output, even though the unweighted output had the better automatic TER.

Interesting ideas

  • I1. Stacking combiners. The N-best output of one combination method is treated as an additional "system" for another combiner. Weak combiners can still add useful variety to a stronger downstream ensemble.
  • I2. TER-based alignment with an MBR skeleton. Hypotheses are aligned by edit distance, and the backbone is the hypothesis closest to all the others under TER. This handles word-order differences that a strict positional alignment cannot, and it could apply to any ensemble of sequence-generating models.
  • I3. Unweighted voting as a strong baseline. Plain consensus decoding with no tuned weights already beat the more elaborate methods. Simple agreement voting is worth checking first before building complex ensembles.
  • I4. Graded agreement levels for phrase confidence. Phrase confidence increases with how strongly systems agree, from exact matches on both source span and position down to merely overlapping target words. This graded notion of agreement could be reused for confidence estimation in other ensemble or annotation-merging settings.
  • I5. Not optimizing one metric blindly. Since it was unclear whether TER or BLEU better tracks the human-edited HTER, the authors watched both and picked the metric to tune for each language. This is a sensible guard whenever automatic proxies for a human judgement disagree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz and Bonnie Dorr. 2007. Combining Outputs from Multiple Machine Translation Systems. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rosti2007combining,
  title = {{Combining Outputs from Multiple Machine Translation Systems}},
  author = {Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz and Bonnie Dorr},
  booktitle = {Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference},
  year = {2007},
  url = {https://aclanthology.org/N07-1029/}
}

References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Srinivas Bangalore, German Bordel, and Giuseppe Riccardi. 2001. Computing consensus translation from multiple machine translation systems. In Proc. ASRU, pages 351–354. find
  2. Richard P. Brent. 1973. Algorithms for Minimization Without Derivatives. Prentice-Hall. find
  3. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proc. ACL, pages 263–270. our card
  4. Jonathan G. Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER). In Proc. ASRU, pages 347–354. find
  5. Robert Frederking and Sergei Nirenburg. 1994. Three heads are better than one. In Proc. ANLP, pages 95– 100. find
  6. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inferences and training of context-rich syntax translation models. In Proc. COL-ING/ACL, pages 961–968. find
  7. Shyamsundar Jayaraman and Alon Lavie. 2005. Multi-engine machine translation guided by explicit word matching. In Proc. EAMT. find
  8. Philipp Koehn. 2004. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In Proc. AMTA. find
  9. Lidia Mangu, Eric Brill, and Andreas Stolcke. 2000. Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Computer Speech and Language, 14(4):373– 400. find
  10. Evgeny Matusov, Nicola Ueffing, and Hermann Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. EACL, pages 33–40. find
  11. Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  12. Franz J. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. ACL, pages 160– 167. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318. our card
  14. Khe Chai Sim, William J. Byrne, Mark J.F. Gales, Hichem Sahbi, and Phil C. Woodland. 2007. Consensus network decoding for statistical machine translation system combination. In Proc. ICASSP. find
  15. Manhung Siu and Herbert Gish. 1999. Evaluation of word confidence for speech recognition systems. Computer Speech and Language, 13(4):299–319. find
  16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciula, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. AMTA. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosti-2007-combining-outputs-multiple-machine-translation/