Elamur Research⌕
☰
home / papers / 2011 / Tuning as Ranking

Tuning as Ranking

Summary

Hopkins and May address a practical bottleneck in statistical MT: MERT, the standard way to set feature weights, becomes unreliable once a model has more than a few dozen features. MIRA-based tuning can handle thousands of features, but it is complex and rarely adopted. The authors propose PRO (pairwise ranking optimization), which keeps MERT's loop of k-best candidate generation and changes only the optimization step. Instead of maximizing the gold score of the single top-ranked output, PRO treats tuning as learning to order candidate translations correctly. Pairs of candidates for the same source sentence become difference vectors labeled by which member scores better under sentence-level BLEU+1. An ordinary linear binary classifier then learns the weights. A synthetic test shows MERT failing to recover known weights as dimensionality grows. Large-scale experiments on phrase-based and syntax-based systems, several language pairs, and baseline and extended feature sets show PRO matching or beating MERT and MIRA. PRO is also more stable across runs. Because it can be added to an existing MERT setup in hours, it lowers the cost of research on rich feature sets.

Key points

  • In a synthetic experiment (500 "sentences", 100 candidates each, dimensionality from 10 to 1000), MERT's line optimization was run with 20 random starts. Its ability to recover a known gold weight vector, measured by cosine similarity, dropped rapidly as dimensionality increased.
  • PRO uses the MegaM classifier and BLEU+1 as the local gold score. The authors report similar results with the WEKA SVM module and the Stanford classifier.
  • In all 12 configurations (system type × language pair × feature set), PRO's test-set results equaled or exceeded those of both MIRA and MERT. The extended features include word-pair features numbering 6,723 in principle.
  • Over five repeated Urdu-English PBMT baseline runs, the standard deviation of final test BLEU was 0.13 for MERT and only 0.05 for PRO.
  • Adoption gap motivating the work: in the ACL 2010 proceedings, 15 MT papers reported tuning with MERT and only one with MIRA.

Interesting ideas

  • I1. Reduce ranking to binary classification on difference vectors. If a scorer is linear, "A should outrank B" becomes a classification problem over the vector x(A) − x(B). This turns any listwise or structured preference task into something a stock classifier can solve. It could be applied to reranking parsers, search results, or candidate normalizations.
  • I2. Change one module and keep the surrounding architecture. The authors left MERT's candidate generation untouched and replaced only the optimizer. This minimal-divergence design greatly eases adoption in existing pipelines, which is a useful principle for introducing any new method into established tooling.
  • I3. Weight smoothing across iterations. Each iteration's classifier is trained from scratch, so the new weights are blended with the previous ones (Ψ = 0.1). This cheap interpolation trick adds stability to any iterative retrain-from-scratch loop.
  • I4. Validate the method on synthetic data with a known answer. Generating data from a hidden gold weight vector and checking whether the optimizer recovers it gives a clean diagnostic of scalability. This kind of test is worth reusing before running costly real experiments.
  • I5. Report stability across repeated runs. Rerunning the same configuration several times and comparing the spread of final scores reveals optimizer variance that a single-run comparison would hide.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

159 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hopkins2011tuning,
  title = {{Tuning as Ranking}},
  author = {Hopkins, Mark and May, Jonathan},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1125/}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, pages 89–96, Bonn, Germany. ACM. find our card
  2. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129–136, Corvalis, OR. find
  3. Samidh Chatterjee and Nicola Cancedda. 2010. Minimum error rate training by sampling the translation lattice. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 606–615, Cambridge, MA, October. Association for Computational Linguistics. find
  4. Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. 2009. Ranking measures and loss functions in learning to rank. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 315–323. find
  5. David Chiang, Steve DeNeefe, Yee Seng Chan, and Hwee Tou Ng. 2008a. Decomposability of translation metrics for improved evaluation and efficient algorithms. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 610–619, Honolulu, HI, October. Association fo… find
  6. David Chiang, Yuval Marton, and Philip Resnik. 2008b. Online large-margin training of syntactic and structural translation features. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 224–233, Honolulu, HI, October. Association for Computational Linguist… find
  7. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 218–226, Boulder, CO, June. Associa… find
  8. David Chiang. 2010. Learning to translate with source and target syntax. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1443–1452, Uppsala, Sweden, July. Association for Computational Linguistics. find our card
  9. Brooke Cowan, Ivona Kuc̆erová, and Michael Collins. 2006. A discriminative model for tree-to-tree translation. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 232–241, Sydney, Australia, July. Association for Computational Linguistics. find
  10. Koby Crammer and Yoram Singer. 2003. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951–991. find
  11. Hal Daumé III. 2004. Notes on CG and LM-BFGS optimization of logistic regression. Paper available at http://pub.hal3.name\#daume04cg-bfgs, implementation available at http://hal3.name/megam/, August. ↗
  12. Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. 2003. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933–969. find
  13. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In HLT-NAACL 2004: Main Proceedings, pages 273–280, Boston, MA, May. Association for Computational Linguistics. find our card
  14. Gloria Goodale. 2008. Language Weaver: fast in translation. The Christian Science Monitor, October 1. http://www.csmonitor.com/Innovation/Tech-Culture/2008/1001/language-weaver-fast-in-translation. ↗
  15. Jan Hajič, Sandra Carberry, Stephen Clark, and Joakim Nivre, editors. 2010. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, July. find
  16. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1). find
  17. Ralf Herbrich, Thore Graepel, and Klaus Obermayer. 1999. Support vector learning for ordinal regression. In Proceedings of the 1999 International Conference on Artificial Neural Networks, pages 97–102. find
  18. Abraham Ittycheriah and Salim Roukos. 2005. A maximum entropy word aligner for Arabic-English machine translation. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 89–96, Vancouver, Canada, October. Association for Compu… find
  19. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  20. Shankar Kumar, Wolfgang Macherey, Chris Dyer, and Franz Och. 2009. Efficient minimum error rate training and minimum bayes-risk decoding for translation hypergraphs and lattices. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference o… find
  21. Alon Lavie and Michael J. Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2–3):105–115, September. find
  22. Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 761–… find
  23. Wolfgang Macherey, Franz Josef Och, Ignacio Thayer, and Jakob Uszkoreit. 2008. Lattice-based minimum error rate training for statistical machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 725– 734, Honolulu, HI, October. Association… find our card
  24. Christopher Manning and Dan Klein. 2003. Optimization, maxent models, and conditional estimation without magic. Tutorial at HLT-NAACL 2003 and ACL 2003. find
  25. I. Dan Melamed, Ryan Green, and Joseph P. Turian. 2003. Precision and recall of machine translation. In Companion Volume of the Proceedings of HLT-NAACL 2003 - Short Papers, pages 61–63, Edmonton, Canada, May–June. Association for Computational Linguistics. find our card
  26. Franz Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 440–447, Hong Kong, October. find our card
  27. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 295–302, Philadelphia, PA, July. Association for Computational Linguistics. find our card
  28. Franz Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. find
  29. Franz Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan, July. Association for Computational Linguistics. find
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. B LEU : a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, July. Association for Computational Linguistic… find
  31. Benjamin Roth, Andrew McCallum, Marc Dymetman, and Nicola Cancedda. 2010. Machine translation using overlapping alignments and samplerank. In Proceedings of Association for Machine Translation in the Americas, Denver, CO. find
  32. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 177–184, Boston, MA, May 2 - May 7. Association for Computational Linguistics. find
  33. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  34. Christoph Tillmann and Tong Zhang. 2005. A localized prediction model for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the ACL, pages 557–564, Ann Arbor, MI, June. Association for Computational Linguistics. find
  35. Wei Wang. 2011. Personal communication. find
  36. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2006. NTT statistical machine translation for IWSLT 2006. In Proceedings of IWSLT 2006, pages 95–102. find
  37. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 764–… find
  38. Deyi Xiong, Qun Liu, and Shouxun Lin. 2006. Maximum entropy based phrase reordering model for statistical machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 521–528, Syd… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/