Elamur Research⌕
☰
home / papers / 2008 / A Discriminative Latent Variable Model for Statistical Machine Translation

A Discriminative Latent Variable Model for Statistical Machine Translation

Paper · 2008 · Phil Blunsom, Trevor Cohn and Miles Osborne

Summary

Discriminative training for statistical machine translation has struggled to beat systems built on heuristic frequency counts. The authors argue that a key reason is spurious ambiguity: a single translation can be produced by a huge number of distinct derivations, and no corpus marks which derivation is correct. They build a globally normalised log-linear model on top of a Hiero-style synchronous context free grammar (SCFG) following Chiang (2007). The model treats the derivation as a latent variable and sums it out both during training and at decoding time. Parameters are estimated by maximising conditional likelihood with a Gaussian prior, using L-BFGS and inside-outside computations over packed derivation forests. On French–English data, choosing the most probable translation gives better results than choosing the single best derivation. With only a very small feature set, the model is competitive with a MERT-tuned Hiero baseline that has no language model. The study also shows that regularisation is necessary, because unregularised likelihood training falls into degenerate solutions.

Key points

  • The number of derivations that yield a reference sentence grows exponentially with sentence length, as shown in Figure 1 on a log scale.
  • The model defines p(e, d|f) with features that decompose over the SCFG rules of a derivation. These features may look at the whole source sentence. The translation probability p(e|f) is obtained by summing over all derivations.
  • Training uses MAP estimation with a zero-mean Gaussian prior and L-BFGS. Gradients come from two packed charts: one holds every derivation of the source, and the other holds only the derivations that reach the reference translation.
  • The grammar is extracted from a corpus of 170k sentences. Only about 130k of these can reproduce their reference, so roughly 40k are dropped before training. The learning curve, measured from 10k to 130k training sentences, was still rising at the largest size.
  • Max-translation decoding beats max-derivation decoding. The likelihood-trained model reaches BLEU comparable to Hiero without a language model, even though it never optimises BLEU directly.

Interesting ideas

  • I1. Marginalise over hidden structure instead of choosing one gold version. When the supervision does not specify the intermediate structure (segmentations, alignments, derivations), summing over every option avoids committing to an arbitrary pseudo-reference. This applies to any task where several latent analyses produce the same output.
  • I2. Learn from many imperfect analyses together. With noisy parallel data there may be no fully correct derivation. Pooling evidence from several partly correct ones lets the model still pick up good fragments, which makes training more robust to noisy corpora.
  • I3. Compute expectations with two charts. One chart is constrained to the reference and one covers all outputs. The gradient is the difference between the feature expectations under these two charts. This pattern transfers directly to latent-variable CRFs and parsers.
  • I4. Treat regularisation as required in conditional-likelihood models. Without a prior, such models can collapse to degenerate solutions, so a prior should be a default design choice whenever features are rich.
  • I5. Run a controlled comparison with the same parameter space. Matching the baseline's features and parameter space isolates the effect of the estimation method itself. This is a clean evaluation design for testing new training objectives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Phil Blunsom, Trevor Cohn and Miles Osborne. 2008. A Discriminative Latent Variable Model for Statistical Machine Translation. Proceedings of ACL-08: HLT.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blunsom2008discriminative,
  title = {{A Discriminative Latent Variable Model for Statistical Machine Translation}},
  author = {Phil Blunsom, Trevor Cohn and Miles Osborne},
  booktitle = {Proceedings of ACL-08: HLT},
  year = {2008},
  url = {https://aclanthology.org/P08-1024/}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Phil Blunsom and Trevor Cohn. 2006. Discriminative word alignment with conditional random fields. In Proc. of the 44th Annual Meeting of the ACL and 21st International Conference on Computational Linguistics (COLING/ACL-2006), pages 65–72, Sydney, Australia, July. find
  2. Marine Carpuat and Dekai Wu. 2007. Improving statistical machine translation using word sense disambiguation. In Proc. of the 2007 Conference on Empirical Methods in Natural Language Processing (EMNLP-2007), pages 61–72, Prague, Czech Republic.
  3. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proc. of the 43rd Annual Meeting of the ACL (ACL-2005), pages 263– 270, Ann Arbor, Michigan, June.
  4. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228.
  5. Stephen Clark and James R. Curran. 2004. Parsing the WSJ using CCG and log-linear models. In Proc. of the 42nd Annual Meeting of the ACL (ACL-2004), pages 103–110, Barcelona, Spain.
  6. Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4).
  7. John DeNero, Dan Gillick, James Zhang, and Dan Klein. 2006. Why generative phrase models underperform surface heuristics. In Proc. of the HLT-NAACL 2006 Workshop on Statistical Machine Translation, pages 31–38, New York City, June. find
  8. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proc. of the 4th International Conference on Human Language Technology Research and 5th Annual Meeting of the NAACL (HLT-NAACL 2004), Boston, USA, May.
  9. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proc. of the 44th Annual Meeting of the ACL and 21st International Conference on Computational Linguistics (…
  10. Abraham Ittycheriah and Salim Roukos. 2007. Direct translation model 2. In Proc. of the 7th International Conference on Human Language Technology Research and 8th Annual Meeting of the NAACL (HLT-NAACL 2007), pages 57–64, Rochester, USA. find
  11. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. of the 3rd International Conference on Human Language Technology Research and 4th Annual Meeting of the NAACL (HLT-NAACL 2003), pages 81–88, Edmonton, Canada, May. find
  12. Philip M. Lewis II and Richard E. Stearns. 1968. Syntax-directed transduction. J. ACM, 15(3):465–488. find
  13. Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proc. of the 44th Annual Meeting of the ACL and 21st International Conference on Computational Linguistics (COLING/ACL-2006), pages 761–768, Sydney, Australia, Ju… find
  14. Robert Malouf. 2002. A comparison of algorithms for maximum entropy parameter estimation. In Proc. of the 6th Conference on Natural Language Learning (CoNLL-2002), pages 49–55, Taipei, Taiwan, August.
  15. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of the 41st Annual Meeting of the ACL (ACL-2003), pages 160–167, Sapporo, Japan. find
  16. Slav Petrov, Adam Pauls, and Dan Klein. 2007. Discriminative log-linear grammars with latent variables. In Advances in Neural Information Processing Systems 20 (NIPS), Vancouver, Canada. find
  17. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In Proc. of the 3rd International Conference on Human Language Technology Research and 4th Annual Meeting of the NAACL (HLT-NAACL 2003), pages 134–141, Edmonton, Canada. find
  18. Christoph Tillmann and Tong Zhang. 2007. A block bigram prediction model for statistical machine translation. ACM Transactions Speech Language Processing, 4(3):6. find
  19. Chao Wang, Michael Collins, and Philipp Koehn. 2007. Chinese syntactic reordering for statistical machine translation. In Proc. of the 2007 Conference on Empirical Methods in Natural Language Processing (EMNLP-2007), pages 737–745, Prague, Czech Republic.
  20. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proc. of the 2007 Conference on Empirical Methods in Natural Language Processing (EMNLP-2007), pages 764–773, Prague, Czech Republic. find
  21. Benjamin Wellington, Joseph Turian, Chris Pike, and I. Dan Melamed. 2006. Scalable purely-discriminative training for word and tree transducers. In Proc. of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA), Boston, USA. find
  22. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377–403.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blunsom-2008-discriminative-latent-variable-model-statistical/