Elamur Research⌕
☰
home / papers / 2007 / Improved Word-Level System Combination for Machine Translation

Improved Word-Level System Combination for Machine Translation

Paper · 2007 · Antti-Veikko Rosti, Spyros Matsoukas and Richard Schwartz

Summary

This paper tackles the problem of merging the outputs of several machine translation systems into one better translation. It builds on confusion network decoding, where hypotheses are aligned to a chosen skeleton and the decoder picks the highest-scoring path. Earlier versions of this approach could produce ungrammatical output and could choose the skeleton from a system that deserved no trust. The authors align hypotheses with TER-style alignment that permits shifts. They build one confusion network per system, using each system's top hypothesis as skeleton, and join all networks into one lattice. Each network gets a prior based on average TER and the system weights. Arcs carry log-posterior scores, so extra features such as language model scores can be combined log-linearly. All weights are tuned automatically with Powell's method against TER, BLEU or METEOR. On the NIST MT05 Arabic-English and Chinese-English tasks, the method beats the authors' earlier combination baseline in BLEU. It also outperforms every individual system when tuned to TER or BLEU. This makes combination more automatic and easier to extend with new features.

Key points

  • Six MT systems were combined. Some outputs were contributed by ISI and the University of Edinburgh. Weights were tuned on NIST MT03+MT04 and results were reported on NIST MT05, for both Arabic→English and Chinese→English.
  • Hypothesis alignment uses a modified Levenshtein alignment with shifts, as in TER. Unlike GIZA++-based alignment, it does not depend on test set size.
  • Against the TER-tuned baseline on Arabic MT05, the new method gains 0.97 BLEU points. On Chinese MT05, the BLEU-tuned combination gains 1.47 BLEU over the baseline.
  • Tuning to METEOR inflates the METEOR score but damages the other metrics. On Arabic MT05, TER rose to about 51.56 and BLEU fell to about 41.73. The cause is many insertions and low precision.
  • TER tuning worked best for Arabic, while BLEU tuning looked better for Chinese.

Interesting ideas

  • I1. One skeleton per system instead of a single chosen skeleton: every system's output serves as a skeleton, and the resulting networks are joined in parallel and weighted by priors. This avoids betting on one ordering and guarantees that a zero-weight system cannot dictate the output. The pattern suits any setting that merges ranked or sequential outputs from several sources.
  • I2. Scoring arcs as log-posteriors rather than raw votes: treating arc scores as log-probabilities lets arbitrary features, such as an LM or a length penalty, be added in one log-linear objective. This idea could transfer to ensembles for tagging, ASR or OCR.
  • I3. Treating the NULL/insertion count as a tunable weight: a single parameter controls how many empty words appear in the output. It acts like a length or insertion penalty and is cheap to add to similar decoders.
  • I4. Metric-agnostic tuning with Powell's method: a derivative-free optimizer lets the same pipeline target any automatic metric. This makes it easy to compare the effect of the tuning objective itself.
  • I5. A cautionary evaluation lesson: a recall-heavy metric that correlates well with humans can still be a poor optimization target, because the tuned system learns to over-generate. Checking several metrics after tuning on one helps reveal this kind of gaming.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Antti-Veikko Rosti, Spyros Matsoukas and Richard Schwartz. 2007. Improved Word-Level System Combination for Machine Translation. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rosti2007improved,
  title = {{Improved Word-Level System Combination for Machine Translation}},
  author = {Antti-Veikko Rosti, Spyros Matsoukas and Richard Schwartz},
  booktitle = {Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics},
  year = {2007},
  url = {https://aclanthology.org/P07-1040/}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. our card
  2. Srinivas Bangalore, German Bordel, and Giuseppe Riccardi. 2001. Computing consensus translation from multiple machine translation systems. In Proc. ASRU, pages 351–354. find
  3. Richard P. Brent. 1973. Algorithms for Minimization Without Derivatives. Prentice-Hall. find
  4. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proc. ACL, pages 263–270. our card
  5. Jonathan G. Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER). In Proc. ASRU, pages 347–354. find
  6. Robert Frederking and Sergei Nirenburg. 1994. Three heads are better than one. In Proc. ANLP, pages 95– 100. find
  7. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inferences and training of context-rich syntax translation models. In Proc. COL-ING/ACL, pages 961–968. find
  8. Shyamsundar Jayaraman and Alon Lavie. 2005. Multi-engine machine translation guided by explicit word matching. In Proc. EAMT, pages 143–152. find
  9. Philipp Koehn. 2004. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In Proc. AMTA, pages 115–124. find
  10. Lidia Mangu, Eric Brill, and Andreas Stolcke. 2000. Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Computer Speech and Language, 14(4):373– 400. find
  11. Evgeny Matusov, Nicola Ueffing, and Hermann Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. EACL, pages 33–40. find
  12. Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  13. Franz J. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. ACL, pages 160– 167. find
  14. Mari Ostendorf, Ashvin Kannan, Steve Austin, Owen Kimball, Richard Schwartz, and Jan Robin Rohlicek. 1991. Integration of diverse recognition methodologies through reevaluation of N-best sentence hypotheses. In Proc. HLT, pages 83–87. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318. our card
  16. Antti-Veikko I. Rosti, Bing Xiang, Spyros Matsoukas, Richard Schwartz, Necip Fazil Ayan, and Bonnie J. Dorr. 2007. Combining outputs from multiple machine translation systems. In Proc. our card
  17. Robert E. Schapire. 1990. The strength of weak learnability. Machine Learning, 5(2):197–227. find
  18. Khe Chai Sim, William J. Byrne, Mark J.F. Gales, Hichem Sahbi, and Phil C. Woodland. 2007. Consensus network decoding for statistical machine translation system combination. In Proc. ICASSP, volume 4, pages 105–108. find
  19. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciula, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. AMTA, pages 223–231. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosti-2007-improved-word-level-system-combination/