Elamur Research
☰
home / papers / 2019 / Training Neural Machine Translation to Apply Terminology Constraints

Training Neural Machine Translation to Apply Terminology Constraints

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1294.pdf

Summary

Translation companies often keep curated term bases that specify how domain terms must be rendered. Neural MT systems need a way to honour these entries at run time. The usual solution is constrained decoding, which modifies the search algorithm. It slows inference and can force terms in awkwardly. Dinu et al. (Amazon AI) instead teach a standard transformer to use terminology given as part of its input. For English→German, they annotate a fraction of the training sentences with target terms in one of two ways: placing the German term after the English one ("train-by-appending") or substituting it ("train-by-replace"). A source-factor stream marks which tokens are ordinary words, source terms or target terms. Term entries come from Wiktionary and IATE. The train and test term lists share no source entries, so evaluation is zero-shot. Compared with the dynamic beam allocation method of Post and Vilar (2018), the trained models used terms more often on IATE, matched or slightly raised BLEU, and decoded as fast as an unconstrained baseline. They also inflect terms when grammar requires it. The approach does not depend on a particular architecture and is cheap enough for production use.

Key points

  • Training setup

    • Data: Europarl plus News Commentary (about 2.2M sentence pairs, WMT 2018 En-De). Roughly 10% extra copies of existing sentences were added with term annotations, so no new parallel data was introduced.
    • Model: a 2-layer encoder / 2-layer decoder transformer in Sockeye, with 32K joint BPE.
    • Source-factor embeddings have size 16 and are concatenated to the subword embeddings.
    • Training sentences were annotated only when the target term actually appeared in the reference. Matching was approximate, at the character-sequence level.

  • Test sets: newstest2017 sentences whose reference contains the term.

    • Wiktionary: 727 sentences, 884 terms.
    • IATE: 414 sentences, 452 terms.

  • Term usage rates (beam 5):

    SystemWiktionaryIATE
    Baseline~76%~76%
    Constrained decoding99.5%82.0%
    Train-by-appending90.7%92.9%
    Train-by-replace93.4%94.5%

  • BLEU and speed:

    • The trained variants gained +0.2 to +0.9 BLEU over the baseline (best: train-by-appending, 26.9 on Wiktionary).
    • Constrained decoding lost up to 0.5 BLEU.
    • Latency was 0.19s for the trained variants versus 0.68s for constrained decoding, which is roughly 3× slower.

  • When target-side matches were approximate (inflected forms):

    • Constrained decoding dropped significantly, by 0.9–1.3 BLEU.
    • Train-by-appending stayed at or above the baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

119 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1294

@inproceedings{dinu2019training,
  title = {{Training Neural Machine Translation to Apply Terminology Constraints}},
  author = {Dinu, Georgiana and Mathur, Prashant and Federico, Marcello and Al-Onaizan, Yaser},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1294},
  url = {https://doi.org/10.18653/v1/p19-1294}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rajen Chatterjee, Matteo Negri, Marco Turchi, Marcello Federico, Lucia Specia, and Frédéric Blain. 2017. Guiding neural machine translation decoding with external knowledge. In Proceedings of the Second Conference on Machine Translation, pages 157–168, Copenhagen, Denmark. Association for Computat… find
  2. Josep Maria Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, Satoshi Enoue, Chiyo Geiss, Joshua Johanson, Ardas Khalsa, Raoum Khiari, Byeongil Ko, Catherine Kobus, Jean Lorieux, Leidiana Martins, Dang-Chua… — [arXiv:1610.05540](https://arxiv.org/abs/1610.05540) find
  3. Post. 2018. The sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 200–207. Association for Machine Translation in the Americas. find
  4. Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1535–1546. find our card
  5. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  6. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. CoRR, abs/1508.04025. find
  7. Ngoc-Quan Pham, Jan Niehues, and Alexander H. Waibel. 2018. Towards one-shot learning for rare-word translation with external experts. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 100–109. find
  8. M. Amin Farajian, Nicola Bertoldi, Matteo Negri, Marco Turchi, and Marcello Federico. 2018. Evaluation of Terminology Translation in Instance-Based Neural MT Adaptation. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pages 149–158, Alicante, Spain.… find
  9. Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orl… find our card
  10. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2017. Search engine guided non-parametric neural machine translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5133–5140, New Orleans, Louisiana, USA. Association for the Advancement of Artificial… find
  11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics. find
  12. Eva Hasler, Adrià Gispert, Gonzalo Iglesias, and Bill Byrne. 2018. Neural machine translation decoding with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pap… find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  14. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt NMT Sockeye train parameters encoder-config: act\_type: relu attention\_heads: 8 conv\_config: null dropout\_act: 0.1 dropout\_attention: 0.1 dropout\_prepost: 0.1 dtype: float32 feed\_forward\_num\_hidden: 2048… find