Elamur Research
☰
home / papers / 2007 / Large Language Models in Machine Translation

Large Language Models in Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D07-1090.pdf

Summary

The paper asks how far n-gram language models can be scaled for statistical machine translation and whether translation quality keeps improving as they grow. The Google authors built a distributed system on MapReduce that handles vocabulary building, n-gram counting and model generation. Sharding by the first two words of each n-gram keeps the workload balanced. The resulting models store smoothed probabilities rather than raw counts and are queried through batched client–server requests. This allows them to be used directly inside a single-pass decoder instead of only for re-ranking candidate translations afterwards. The authors also propose Stupid Backoff, an unnormalized scoring scheme that uses plain relative frequencies with a fixed backoff factor and is cheap to compute at massive scale. Using up to 2 trillion training tokens, they trained Arabic–English systems and charted BLEU learning curves. Quality rose roughly linearly with each doubling of data, even at the largest sizes, and Stupid Backoff caught up with Kneser-Ney smoothing once data became large. The work showed that sheer data volume, paired with simple estimators, can beat more sophisticated smoothing.

Key points

  • Training data reached up to 2 trillion tokens, yielding 5-gram models with up to 300 billion n-grams. That is about two orders of magnitude beyond earlier distributed LMs, which used 3–4 billion words.
  • Stupid Backoff uses α = 0.4 without discounting or normalization. Its scores are not true probabilities, which is acceptable because the log-linear decoder relies only on relative feature values.
  • The experiments used four LM data sources (target, ldcnews, webnews, web) with an Arabic–English system. The full version of this system scored 0.4535 BLEU at NIST 2006.
  • On target data, gains per doubling were about 0.62 BP for Kneser-Ney and 0.66 BP for Stupid Backoff. The gap between the two methods disappeared on larger sets. The web-scale Stupid Backoff model beat the best Kneser-Ney model by 0.82 BP.
  • Test-set 5-gram coverage grew from 0.06 with 13 million tokens to 0.56 with 2 trillion tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 161 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{brants2007large,
  title = {{Large Language Models in Machine Translation}},
  author = {Brants, Thorsten and Popat, Ashok C. and Xu, Peng and Och, Franz J. and Dean, Jeffrey},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1090/}
}

References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  2. Stanley F. Chen and Joshua Goodman. 1998. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard, Cambridge, MA, USA. find
  3. Jeffrey Dean and Sanjay Ghemawat. 2004. Mapreduce: Simplified data processing on large clusters. In Sixth Symposium on Operating System Design and Implementation (OSDI-04), San Francisco, CA, USA. find
  4. Ahmad Emami, Kishore Papineni, and Jeffrey Sorensen. 2007. Large-scale distributed language modeling. In Proceedings of ICASSP-2007, Honolulu, HI, USA. find
  5. Joshua Goodman. 2001. A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft Research, Redmond, WA, USA. find
  6. Frederick Jelinek and Robert L. Mercer. 1980. Interpolated estimation of Markov source parameters from sparse data. In Pattern Recognition in Practice, pages 381–397. North Holland. find
  7. Slava Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3). find
  8. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 181– 184. find
  9. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP-04, Barcelona, Spain. find
  10. Hermann Ney and Stefan Ortmanns. 1999. Dynamic programming search for continuous speech recognition. IEEE Signal Processing Magazine, 16(5):64–83. find
  11. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses. John Wiley & Sons. find
  12. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL-02, pages 311–318, Philadelphia, PA, USA. find our card
  14. Ying Zhang, Almut Silja Hildebrand, and Stephan Vogel. 2006. Distributed language modeling for n-best list re-ranking. In Proceedings of EMNLP-2006, pages 216–223, Sydney, Australia. find