Elamur Research
☰
home / papers / 2016 / Linguistic Input Features Improve Neural Machine Translation

Linguistic Input Features Improve Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2209.pdf

Summary

Neural machine translation (NMT) learns from raw parallel text, so it is unclear whether explicit linguistic annotation still helps. Sennrich and Haddow test this by extending the encoder of an attentional encoder–decoder model (Bahdanau-style, built on the dl4mt code and released as Nematus) so that each source token can carry several features. Each feature gets its own embedding, and the embeddings are concatenated into a vector whose length equals the original embedding size. The rest of the network is unchanged. The source-side features are lemmas, subword-position tags, morphological descriptions, POS tags and dependency labels. All are predicted automatically and copied onto every BPE fragment of a word. The authors train English↔German systems on WMT16 data and English→Romanian systems, then compare perplexity, BLEU and CHRF3 against plain baselines. Adding the features generally lowers perplexity and raises translation scores, including when synthetic back-translated data is used. The work shows that annotation remains useful for strong neural models and offers a simple, general way to add it.

Key points

  • Data and setup:
    • English↔German training used about 4.2M WMT16 sentence pairs with joint BPE (89,500 merges).
    • newstest2013 served for validation; newstest2015 and newstest2016 were the test sets.
  • Annotation tools and subword tags:
    • English input was annotated with Stanford CoreNLP.
    • German input was annotated with ParZu, with lemmas from Zmorge.
    • Subword tags mark each BPE unit as word-beginning (B), inside (I), end (E), or a complete word (O).
  • English→German (parallel data only): combining all features raised BLEU from 23.0 to 23.8 (test15) and from 27.8 to 28.4 (test16). In single-feature tests, POS tags gave the largest gain, reaching 24.0 and 28.9.
  • With synthetic training data:
    • German→English BLEU improved from 37.5 to 38.5 on newstest2016.
    • English→German BLEU barely changed, though perplexity still dropped (49.7 to 48.4).
  • English→Romanian:
    • Features added about 1.0 BLEU in both settings: 23.8→24.8 with parallel data only, and 28.2→29.2 with synthetic data.
    • The best system was competitive with the top WMT16 submission (28.9).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Part-of-speech tagging · Machine translation

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 153 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2209 · arXiv 1606.02892

@inproceedings{sennrich2016linguistic,
  title = {{Linguistic Input Features Improve Neural Machine Translation}},
  author = {Sennrich, Rico and Haddow, Barry},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2209},
  eprint = {1606.02892},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w16-2209}
}

References

4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. form multi-source translation (Zoph and Knight, 2016). The main technical difference is that in our approach, the encoder and attention layers are shared between features, which we deem appropriate for the types of features that we tested. find
  2. Andrei Alexandrescu and Katrin Kirchhoff. 2006. Factored Neural Language Models. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 1–4, New York City, USA. Association for Computational Linguistics. \#\# Conclusion Dzmitry Bahdanau, Kyunghyu… find
  3. In this paper we investigate whether linguistic input features are beneficial to neural machine translation, and our empirical evidence suggests that this is the case. find
  4. Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics. find our card