Elamur Research⌕
☰
home / papers / 2007 / Factored Translation Models

Factored Translation Models

Summary

Koehn and Hoang address a limitation of standard phrase-based statistical machine translation. It treats every surface word form as an unrelated token and ignores linguistic knowledge, which causes data sparseness, especially for morphologically rich languages. They generalise the phrase-based framework so that each word becomes a vector of factors, which can include the word as written, its base form, its grammatical category, its morphological properties, or classes induced automatically. Translating a phrase is decomposed into a series of mapping steps. Translation steps map input factors to output factors at the phrase level. Generation steps derive further output factors within each output word. All components, including n-gram language models over any factor, are combined in a log-linear model, with weights tuned by minimum error rate training.

The authors test the framework on German–English (News Commentary), English–Chinese and Chinese–English (IWSLT 2006). It yields BLEU gains of up to about 2 points, plus better grammatical coherence. Plain phrase-based translation turns out to be a special case of the factored model, so the approach offers a unified way to add word-level annotation without separate pre- or post-processing pipelines.

Key points

  • On German–English with the News Commentary corpus, the surface baseline scored 18.19% BLEU and adding a POS language model raised it to 19.05%. A lemma/morphology-only model fell to 14.46%. A backoff (alternative path) model combining surface and lemma/morph tables reached 19.47%.
  • In that test set of 26,898 words, there were 3276 unknown word forms but only 2589 unknown lemmas. The lemma/morph path could therefore handle 687 extra words.
  • On English–Chinese (IWSLT 2006, 39,953 sentences), output-side automatic word classes used with a 7-gram sequence model raised BLEU from 19.54% to 21.10%.
  • Integrated recasing for Chinese–English (IWSLT 2006) beat the usual two-pass approach (SMT followed by recasing), scoring 21.08% vs. 20.65% BLEU.
  • Training reuses symmetrized GIZA++ alignments. Generation tables are estimated from the target side only, so monolingual data can be used for them. The framework was also applied to CCG supertags, domain adaptation and English–Czech translation.

Interesting ideas

  • I1. Words as factor vectors. Representing each token as a bundle of parallel annotations lets one system combine linguistic tags and unsupervised clusters. This is useful for any sequence task where raw forms are sparse, such as Ukrainian morphology.
  • I2. Splitting translation into translate-then-generate steps. Lemmas and morphological tags are mapped separately, and target forms are then generated from them. This mirrors analysis/synthesis pipelines and pools evidence across inflected forms of the same lemma.
  • I3. Alternative paths instead of hard backoff. The surface-form table and the decomposed lemma/morph route are kept as competing components, and tuned weights decide which to trust. This avoids throwing away surface information while still covering rare or unseen forms, a pattern applicable to many OOV-handling problems.
  • I4. Sequence models over any factor. High-order n-gram models (7- or 9-gram) are cheap to build over small tag vocabularies like POS or word classes, and they can enforce agreement and fluency. This lightweight trick transfers to generation tasks beyond MT.
  • I5. Folding post-processing into the model. Recasing becomes just another generation step rather than a separate pass. Other cleanup stages, such as normalisation or diacritic restoration, might likewise benefit from being optimised jointly instead of in a pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

203 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 190 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{koehn2007factored,
  title = {{Factored Translation Models}},
  author = {Koehn, Philipp and Hoang, Hieu},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1091/}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alshawi, H., Bangalore, S., and Douglas, S. (1998). Automatic acquisition of hierarchical transduction models for machine translation. In Proceedings of the 36th Annual Meeting of the Association of Computational Linguistics (ACL). find
  2. Birch, A., Osborne, M., and Koehn, P. (2007). CCG supertags in factored statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 9–16, Prague, Czech Republic. Association for Computational Linguistics. find
  3. Bojar, O. (2007). English-to-Czech factored machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 232–239, Prague, Czech Republic. Association for Computational Linguistics. find
  4. Brill, E. (1995). Transformation-based error-driven learning and natural language processing: A case study in part of speech tagging. Computational Linguistics, 21(4). find our card
  5. Collins, M., Koehn, P., and Kucerova, I. (2005). Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 531–540, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  6. Galley, M., Graehl, J., Knight, K., Marcu, D., DeNeefe, S., Wang, W., and Thayer, I. (2006). Scalable inference and training of context-rich syntactic translation models. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for C… find our card
  7. Koehn, P., Federico, M., Shen, W., Bertoldi, N., Hoang, H., Callison-Burch, C., Cowan, B., Zens, R., Dyer, C., Bojar, O., Moran, C., Constantin, A., and Herbst, E. (2006). Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. Technical r… find
  8. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the Annual Meeting of the Associati… find
  9. Koehn, P. and Knight, K. (2003). Feature-rich translation of noun phrases. In 41st Annual Meeting of the Association of Computational Linguistics (ACL). find
  10. Koehn, P. and Monz, C. (2006). Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121, New York City. Association for Computational Linguistics. find our card
  11. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase based translation. In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). find
  12. Koehn, P. and Schroeder, J. (2007). Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 224–227, Prague, Czech Republic. Association for Computational Linguistics. find our card
  13. Lee, Y.-S. (2004). Morphological analysis for statistical machine translation. In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). find
  14. Lita, L. V., Ittycheriah, A., Roukos, S., and Kambhatla, N. (2003). tRuEcasIng. In Hinrichs, E. and Roth, D., editors, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 152–159. find
  15. Melamed, I. D. (2004). Statistical machine translation by parsing. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 653–660, Barcelona, Spain. find
  16. Menezes, A. and Quirk, C. (2005). Microsoft research treelet translation system: IWSLT evaluation. In Proc. of the International Workshop on Spoken Language Translation. find
  17. Nießen, S. and Ney, H. (2001). Toward hierarchical models for statistical machine translation of inflected languages. In Workshop on Data-Driven Machine Translation at 39th Annual Meeting of the Association of Computational Linguistics (ACL), pages 47–54. find
  18. Och, F. J. (2003). Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL). find
  19. Och, F. J., Gildea, D., Khudanpur, S., Sarkar, A., Yamada, K., Fraser, A., Kumar, S., Shen, L., Smith, D., Eng, K., Jain, V., Jin, Z., and Radev, D. (2004). A smorgasbord of features for statistical machine translation. In Proceedings of the Joint Conference on Human Language Technologies and the An… find
  20. Sadat, F. and Habash, N. (2006). Combination of arabic preprocessing schemes for statistical machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 1–8, Sydney, Australia. A… find
  21. Schmidt, H. and Schulte im Walde, S. (2000). Robust German noun chunking with a probabilistic context-free grammar. In Proceedings of the International Conference on Computational Linguistics (COLING). find
  22. Shen, W., Zens, R., Bertoldi, N., and Federico, M. (2006). find
  23. The JHU Workshop 2006 IWSLT System. In Proc. of the International Workshop on Spoken Language Translation, pages 59–63, Kyoto, Japan. find
  24. Talbot, D. and Osborne, M. (2006). Modelling lexical redundancy for machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 969–976, Sydney, Australia. Association for Comput… find
  25. Wang, W., Knight, K., and Marcu, D. (2006). Capitalizing machine translation. In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). find
  26. Wu, D. (1997). Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3). find our card
  27. Yamada, K. and Knight, K. (2001). A syntax-based statistical translation model. In Proceedings of the 39th Annual Meeting of the Association of Computational Linguistics (ACL). find our card
  28. Yang, M. and Kirchhoff, K. (2006). Phrase-based backoff models for machine translation of highly inflected languages. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/koehn-2007-factored-translation-models/