Elamur Research⌕
☰
home / papers / 2016 / Multi-Source Neural Translation

Multi-Source Neural Translation

Summary

The paper tackles translation when the same sentence is already available in two source languages, for example when a document has been translated once by hand and must then be rendered into further languages. Earlier multi-source MT combined outputs of separately trained bilingual systems. Zoph and Knight instead train one neural model, P(e|f, g), directly on trilingual data. Each source language gets its own four-layer LSTM encoder, and the final encoder states are merged before decoding. They test two merging schemes: a Basic concatenate-and-project combiner and a Child-Sum combiner inspired by Tree-LSTMs. They also extend local attention so the decoder attends to both encoders at once. The data is a trilingual subset of WMT 2014. For French+German→English, the best multi-source attention model gains +4.8 BLEU over a strong single-source attention baseline. Gains are much smaller when the sources are English and French and the target is German. This suggests that more dissimilar source languages resolve each other's ambiguities better. The work shows that "triangulation" can be learned end-to-end inside a single network.

Key points

  • Training data comes from WMT 2014 French/English/German: about 2.38M aligned segment triples. Development uses 3000 WMT sentences, and evaluation is done on 1503 lines drawn from the WMT test set that are available in all three languages. BLEU is case-sensitive, single-reference, computed with Moses multi-bleu.perl.
  • The architecture uses four-layer LSTM encoder-decoders with 1000-dimensional hidden states. The attention baseline follows Luong et al. (2015b) local-p attention with input feeding.
  • For French+German→English, Basic attention reaches 30.0 BLEU versus 25.2 for French-only attention (+4.8). Child-Sum attention reaches 29.6 (+4.4). Without attention, Basic improves 21.0→23.2 and Child-Sum improves 21.0→22.5.
  • Feeding two copies of the same French sentence gives no gain (20.7 BLEU). This shows the improvement comes from the second language, not from extra model capacity.
  • For French+English→German, gains are at most +1.1 BLEU (e.g., 17.6→18.6 with attention). The code is released as Zoph_RNN (github.com/isi-nlp).

Interesting ideas

  • I1. Merge cell states by summing them instead of projecting them. The authors report that concatenating and linearly transforming cell states made training diverge because of large values, while simple addition worked. This is a practical hint for anyone fusing several LSTM encoders.
  • I2. Give each input its own forget gate (Child-Sum style). A learned gate per encoder lets the model decide how much to trust each source. This could transfer to fusing parallel inputs such as dialect variants, OCR versions, or multimodal signals.
  • I3. Use separate attention per source, concatenated before output. Each encoder keeps its own attention parameters and context vector, and both vectors are joined with the decoder state. This is a simple way to add any number of aligned sources to an attention model.
  • I4. Control for extra capacity by duplicating the input. Running the multi-source model on two copies of the same source separates the true benefit of a second language from the effect of simply having more parameters. This control is worth reusing in any multi-input experiment.
  • I5. Choose the second source language for its distance from the first. The results suggest that a more distant second language adds more disambiguating information. For pipelines with existing human translations (e.g., Ukrainian alongside a typologically different language), the choice of auxiliary language may matter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

132 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 120 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1004

@inproceedings{zoph2016multi,
  title = {{Multi-Source Neural Translation}},
  author = {Zoph, Barret and Knight, Kevin},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1004},
  url = {https://doi.org/10.18653/v1/n16-1004}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, and L. Specia, editors. 2014. Proc. of the Ninth Workshop on Statistical Machine Translation. Associ… find our card
  2. C. Callison-Burch. 2002. Co-training for statistical machine translation. Master’s thesis, School of Informatics, University of Edinburgh. M. A. Castaño and F. Casacuberta. 1997. A connectionist approach to machine translation. In EU-ROSPEECH. find
  3. A. Deri and K. Knight. 2015. How to make a Frenemy: Multitape FSTs for portmanteau generation. In Proc. NAACL. D. Dong, H. Wu, W. he, D. Yu, and H. Wang. 2015. Multi-task learning for multiple language translation. In Proc. find
  4. ACL. C. Elgot and J. Mezei. 1965. On relations defined by generalized finite automata. IBM Journal of Research and Development, 9(1):47–68. find
  5. W. A Gale and K. W Church. 1993. A program for aligning sentences in bilingual corpora. Computational linguistics, 19(1):75–102. J. González-Rubio and F. Casacuberta. 2010. On the use of median string for multi-source translation. In Proc. find our card
  6. ICPR. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8). find
  7. R. Kaplan and M. Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20(3):331–378. find
  8. M. Kay. 2000. Triangulation in translation. Keynote at MT 2000 Conference, University of Exeter. find
  9. M. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser. 2015a. Multi-task sequence to sequence learning. In arXiv. http://arxiv.org/abs/1511.06114. ↗ our card
  10. M. Luong, H. Pham, and C. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proc. find
  11. EMNLP. E. Matusov, N. Ueffing, and H. Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. EACL. find
  12. A. Max, J. Crego, and F. Yvon. 2010. Contrastive lexical evaluation of machine translation. In Proc. find
  13. LREC. R. Neco and M. Forcada. 1997. Asynchronous translations with recurrent neural nets. In International Conf. on Neural Networks, volume 4, pages 2535–2540. find
  14. F. J. Och and H. Ney. 2001. Statistical multi-source translation. In Proc. find
  15. MT Summit. J. Schroeder, T. Cohn, and P. Koehn. 2009. Word lattices for multi-source translation. In Proc. find
  16. EACL. L. Schwartz. 2008. Multi-source translation methods. find
  17. Proc. AMTA. M. Simard. 1999. Text-translation alignment: Three languages are better than two. In Proc. EMNLP/VLC. find
  18. I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. find
  19. NIPS. K. S. Tai, R. Socher, and C. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. find
  20. ACL. P. J. Werbos. 1990. Backpropagation through time: what it does and how to do it. Proc. IEEE, 78(10):1550– 1560. find
  21. W. Zaremba, I. Sutskever, and O. Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/