Elamur Research⌕
☰
home / papers / 2008 / Online Large-Margin Training of Syntactic and Structural Translation Features

Online Large-Margin Training of Syntactic and Structural Translation Features

Paper · 2008 · David Chiang, Yuval Marton and Philip Resnik

Summary

Statistical machine translation systems are usually tuned with MERT. It reliably optimizes only a few dozen feature weights: by informal estimates, between 15 and 30. This makes it hard to add new features to the model. The authors build on the work of Watanabe et al. and propose online large-margin training with the MIRA algorithm in place of MERT. Training is distributed across several processors that exchange updates. The method makes broader use of the decoder's parse forest. The oracle translation is selected as a trade-off between BLEU and the model score. In addition, negative examples are added: translations that the model mistakenly scores too highly. With the same feature set, MIRA is on par with MERT in both quality and computational cost. The method is then applied to the hierarchical phrase-based model Hiero with two groups of features. The first group consists of the soft syntactic constraints of Marton and Resnik, which are now all trained simultaneously. The second is a new structural reordering model that takes constituent length into account. Each group on its own yields a statistically significant gain. Together they have 56 weights and improve the result by 2.6 BLEU on the Arabic-English NIST 2006 data. The work shows that, freed from the limitations of MERT, much richer translation models can be built.

Key points

  • Task and data: Arabic-to-English translation, with Hiero as the baseline system. Testing was carried out on a portion of the NIST 2006 set, covering the newswire and newsgroups genres.
  • Results (Table 1): the baseline system with MERT has 12 features and 44.6 BLEU on the full test set. MIRA with the same features gives 45.3. MIRA with the combination of syntactic and distortion features (56 weights) gives 47.2. On the dev set the result rose from 52.0 to 53.6. MERT was unable to take advantage of the more detailed (fine-grained) features, whereas MIRA was.
  • Method: the loss function is based on BLEU. BLEU is computed in the context of a pseudo-document, an exponentially weighted moving average of previous translations (coefficient 0.9). The oracle is selected with the parameter µ = 0.5. Negative examples are taken with µ = ∞, i.e., via loss-augmented inference. The regularization parameter is C = 0.01.
  • Ablation (Table 2): if the additional negative examples are removed, training breaks down: the models generate translations that are too long. When the processors do not exchange updates, the dev result drops from 53.6 to 53.1. Standard local updating gives 0.5 BLEU less than the proposed oracle selection.
  • New features: the soft syntactic constraints reward or penalize a derivation depending on how well it agrees with the syntactic parse of the source sentence. The structural reordering model addresses a known gap in Hiero: it models how word order depends on constituent length.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,087 works

Cite

Cite as David Chiang, Yuval Marton and Philip Resnik. 2008. Online Large-Margin Training of Syntactic and Structural Translation Features. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chiang2008online,
  title = {{Online Large-Margin Training of Syntactic and Structural Translation Features}},
  author = {David Chiang, Yuval Marton and Philip Resnik},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  doi = {10.3115/1613715.1613747},
  url = {https://doi.org/10.3115/1613715.1613747}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abhishek Arun and Philipp Koehn. 2007. Online learning methods for discriminative training of phrase based statistical machine translation. In Proc. MT Summit XI. find
  2. Phil Blunsom, Trevor Cohn, and Miles Osborne. 2008. A discriminative latent variable model for statistical machine translation. In Proc. ACL-08: HLT.
  3. Fabiano C. Botelho, Yoshiharu Kohayakawa, and Nivio Ziviani. 2005. A practical minimal perfect hashing method. In 4th International Workshop on Efficient and Experimental Algorithms (WEA05). find
  4. Daniel Cer, Daniel Jurafsky, and Christopher D. Manning. 2008. Regularization and search for minimum error rate training. In Proc. Third Workshop on Statistical Machine Translation. find
  5. Colin Cherry. 2008. Cohesive phrase-based decoding for statistical machine translation. In Proc. ACL-08: HLT. find
  6. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proc. ACL 2005.
  7. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2). ↗
  8. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In Proc. EMNLP 2002.
  9. Koby Crammer and Yoram Singer. 2003. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951–991. find
  10. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7:551–585. find
  11. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic tagging of Arabic text: From raw text to base phrase chunks. In Proc. HLT/NAACL 2004. Companion volume.
  12. Markus Dreyer, Keith Hall, and Sanjeev Khudanpur. 2007. Comparing reordering constraints for SMT using efficient B oracle computation. In Proc. 2007 Workshop on Syntax and Structure in Statistical Translation. find
  13. Kevin Duh and Katrin Kirchoff. 2008. Beyond log-linear models: Boosted minimum error rate training for nbest re-ranking. In Proc. ACL-08: HLT, Short Papers. find
  14. Yoav Freund and Robert E. Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37:277–296. ↗
  15. Dan Klein and Chris D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS 2002). find
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. HLT-NAACL 2003. ↗
  17. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. EMNLP 2004.
  18. Gregor Leusch, Evgeny Matusov, and Hermann Ney. 2008. Complexity of finding the BLEU-optimal hypothesis in a confusion network. In Proc. EMNLP 2008. This volume. find
  19. Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proc. COLING-ACL 2006. find
  20. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In Proc. COLING 2004.
  21. Yuval Marton and Philip Resnik. 2008. Soft syntactic constraints for hierarchical phrased-based translation. In Proc. ACL-08: HLT.
  22. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proc. ACL 2005.
  23. Haitao Mi, Liang Huang, and Qun Liu. 2008. Forest-based translation. In Proc. ACL-08: HLT.
  24. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proc. ACL 2002.
  25. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. ACL 2003. find
  26. John C. Platt. 1998. Fast training of support vector machines using sequential minimal optimization. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods: Support Vector Learning, pages 195–208. MIT Press. ↗
  27. David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proc. COLING/ACL 2006, Poster Sessions.
  28. David Talbot and Thorsten Brants. 2008. Randomized language models via perfect hash functions. In Proc. ACL-08: HLT. find
  29. Ben Taskar, Dan Klein, Michael Collins, Daphne Koller, and Christopher Manning. 2004. Max-margin parsing. In Proc. EMNLP 2004, pages 1–8. find
  30. Ben Taskar, Vassil Chatalbashev, Daphne Koller, and Carlos Guestrin. 2005. Learning structured prediction models: A large margin approach. In Proc. ICML 2005. find
  31. Christoph Tillmann and Tong Zhang. 2006. A discriminative global training algorithm for statistical MT. In Proc. COLING-ACL 2006. find
  32. Joseph Turian, Benjamin Wellington, and I. Dan Melamed. 2007. Scalable discriminative learning for natural language parsing and translation. In Advances in Neural Information Processing Systems 19 (NIPS 2006). ↗
  33. Taro Watanabe, Jun Suzuki, Hajime Tsukuda, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proc. EMNLP 2007. find
  34. Dekai Wu. 1996. A polynomial-time algorithm for statistical machine translation. In Proc. 34th Annual Meeting of the Association for Computational Linguistics, pages 152–158.
  35. Min Zhang, Hongfei Jiang, Aiti Aw, Haizhou Li, Chew Lim Tan, and Sheng Li. 2008. A tree sequence alignment-based tree-to-tree translation model. In Proc. ACL-08: HLT. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chiang-2008-online-large-margin-training-syntactic/