Elamur Research⌕
☰
home / papers / 2002 / Discriminative Training and Maximum Entropy Models for Statistical Machine Translation

Discriminative Training and Maximum Entropy Models for Statistical Machine Translation

Summary

Classical statistical MT follows the source-channel scheme. A target language model and an inverse translation model are trained separately and multiplied with equal weight. That combination is only optimal if both models are exact, and it gives no clean way to add more knowledge. Och and Ney instead model the posterior probability of the translation directly with a log-linear (maximum entropy) model. Every knowledge source becomes a weighted feature function of the source sentence, the target sentence and hidden variables such as alignments. The source-channel model is the special case where two log-probability features both have weight 1. They build this on the alignment template system and train the weights with GIS. The normalization sum is approximated over n-best lists, and the criterion is adapted to several reference translations. Tuning only the weights of the four baseline features already lowers every error measure. Adding a word penalty, a class-based five-gram LM and a conventional dictionary feature improves results further. The framework turns MT into an extensible feature-combination problem and became the basis of later log-linear SMT systems.

Key points

  • Baseline features are a word trigram LM plus three alignment template components: the template alignment model, the template model and the lexicon model.
  • Moving from equal weights to trained weights reduced SER from 86.9 to 81.7, WER from 42.8 to 40.2, and raised BLEU from 43.9 to 49.7.
  • The full system (ME+WP+CLM+MX) reached SER 77.8, WER 38.4, mWER 31.9, BLEU 55.2, SSER 28.8 and IER 30.9.
  • GIS converged after roughly 4000 iterations, with no significant overfitting on the test set.
  • The learned scaling factors show large weights for the word penalty (about 2.6–3.0) and the conventional dictionary (2.92). Adding a new feature also shifted the weights of the existing ones.

Interesting ideas

  • I1. Treat existing component models as log-features with learned weights. Any pipeline that multiplies separately trained probability models can gain from tuning their relative weights. This holds for ASR, OCR post-correction and tagging alike.
  • I2. Let a curated resource earn its weight. A feature that counts matches with a hand-made dictionary lets training decide how much to trust it compared with automatically learned lexicons. This is a cheap way to inject manually built lexical resources into a statistical system.
  • I3. Split features by how they fit into search. Features that decompose locally, such as the word penalty and n-gram LMs, go into dynamic-programming search. Global ones, such as dictionary counts, are applied only when rescoring n-best lists. This keeps decoding tractable while still allowing rich features.
  • I4. Approximate an intractable normalizer with an n-best list. Summing over a set of highly probable hypotheses instead of all outputs makes discriminative training feasible for structured outputs.
  • I5. Evaluate with both automatic and human-judged measures. Reporting SER, WER, PER, mWER and BLEU next to SSER and IER shows whether automatic gains match judged translation quality. This is a useful habit when evaluating any generation system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

427 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 410 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073083.1073133

@inproceedings{och2002discriminative,
  title = {{Discriminative Training and Maximum Entropy Models for Statistical Machine Translation}},
  author = {Och, Franz Josef and Ney, Hermann},
  booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  doi = {10.3115/1073083.1073133},
  url = {https://doi.org/10.3115/1073083.1073133}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer. 1986. Maximum mutual information estimation of hidden markov model parameters. In Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, pages 49–52, Tokyo, Japan, April. find
  2. A. L. Berger, S. A. Della Pietra, and V. J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–72, March. find our card
  3. P. Beyerlein. 1997. Discriminative model combination. In Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding, pages 238– 245, Santa Barbara, CA, December. find
  4. P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  5. J. N. Darroch and D. Ratcliff. 1972. Generalized iterative scaling for log-linear models. Annals of Mathematical Statistics, 43:1470–1480. find
  6. B. H. Juang, W. Chou, and C. H. Lee. 1995. Statistical and discriminative methods for speech recognition. In A. J. R. Ayuso and J. M. L. Soler, editors, Speech Recognition and Coding - New Advances and Trends. Springer Verlag, Berlin, Germany. find
  7. H. Ney. 1995. On the probabilistic-interpretation of neural-network classifiers and discriminative training criteria. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(2):107–119, February. find
  8. S. Nießen, F. J. Och, G. Leusch, and H. Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proc. of the Second Int. Conf. on Language Resources and Evaluation (LREC), pages 39–45, Athens, Greece, May. find our card
  9. F. J. Och, C. Tillmann, and H. Ney. 1999. Improved alignment models for statistical machine translation. In Proc. of the Joint SIGDAT Conf. on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 20–28, University of Maryland, College Park, MD, June. find our card
  10. K. A. Papineni, S. Roukos, and R. T. Ward. 1997. Feature-based language understanding. In European Conf. on Speech Communication and Technology, pages 1435–1438, Rhodes, Greece, September. find
  11. K. A. Papineni, S. Roukos, and R. T. Ward. 1998. Maximum likelihood and discriminative training of direct translation models. In Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, pages 189–192, Seattle, WA, May. find
  12. K. A. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division, Thomas J. Watson Research Center, Yorktown Heights, NY, September. find our card
  13. J. Peters and D. Klakow. 1999. Compact maximum entropy language models. In Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding, Keystone, CO, December. find
  14. R. Schlüter and H. Ney. 2001. Model-based MCE bound to the true Bayes’ error. IEEE Signal Processing Letters, 8(5):131–133, May. find
  15. W. Wahlster. 1993. Verbmobil: Translation of face-to-face dialogs. In Proc. of MT Summit IV, pages 127– 135, Kobe, Japan, July. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/och-2002-discriminative-training-maximum-entropy-models/