Elamur Research
☰
home / papers / 2017 / How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/E17-2060.pdf

Summary

Standard metrics such as BLEU reward local n-gram overlap. They therefore say little about whether a translation system gets specific grammatical or semantic phenomena right. This matters for character-level NMT decoders, whose long output sequences might weaken sentence-level coherence. Sennrich proposes using the NMT model's own scoring ability. A human reference translation is paired with a minimally altered copy that contains one deliberate error, and the test checks whether the model gives the reference a higher (length-normalized) probability.

From the WMT English→German newstest sets, rules automatically generated roughly 97 000 such pairs, using:

  • training-corpus statistics,
  • the ParZu parser,
  • a finite-state morphology.

The pairs cover five error categories and form the released resource LingEval97. Three systems were trained on WMT15 data (BPE-to-BPE, BPE-to-char and char-to-char) and compared against the WMT16 winning system. BLEU scores for the three were close. Even so, the character-level decoders handled unseen names better but were weaker on agreement and on verb particles. This shows a trade-off that BLEU does not reveal, and the evaluation method is reproducible and works with any model.

Key points

  • LingEval97 contains about 97 000 contrastive pairs built from 22 191 newstest sentences. The categories are:
    • NP agreement (21 813 pairs)
    • subject-verb agreement (35 105)
    • separable verb particle (2 450)
    • polarity insertion/deletion (22 760 / 4 043)
    • transliteration of unseen names (3 490)
  • BLEU on newstest2014 (detokenized NIST) was 20.1 for BPE-to-BPE, 19.4 for BPE-to-char and 19.7 for char-to-char.
  • Grammar: BPE-to-BPE scored higher than char-to-char on NP agreement (95.6% vs 93.9%), subject-verb agreement (93.4% vs 91.5%) and verb particles (91.1% vs 86.7%).
  • Transliteration: the character-level decoders scored about 98.3–98.6%, compared with 96.1% for BPE-to-BPE.
  • Negation: deleting a negation marker was the hardest error type for all models (88.4–92.7%). The human ceilings, estimated on sampled failure cases, ranged from 98.5% to 99.9%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

109 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 98 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1612.04629

@inproceedings{sennrich2017how,
  title = {{How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs}},
  author = {Sennrich, Rico},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.18653/v1/e17-2060},
  eprint = {1612.04629},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/e17-2060}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sharid Loáiciga and Kristina Gulordava. 2016. Discontinuous Verb Phrases in Parsing and Machine Translation of English and German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  3. Minh-Thang Luong and D. Christopher Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063. Association for Computational Linguistics. find our card
  4. Boxing Chen and Xiaodan Zhu. 2014. Bilingual Sentiment Consistency for Statistical Machine Translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 607–615, Gothenburg, Sweden, April. Association for Computational Linguistics. find
  5. Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In 18th Int. Conf. on Computational Linguistics, pages 1081–1085. find
  6. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A Character-level Decoder without Explicit Segmentation for Neural Machine Translation. CoRR, abs/1603.06147. ↗
  7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, PA. Association for Computational Linguistics. find our card
  8. Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn. 2014. Integrating an Unsupervised Transliteration Model into Statistical Machine Translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, pages 148–153, Gothenburg, Sweden. find
  9. Maja Popovic. 2011. Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output. Prague Bull. Math. Linguistics, 96:59–68. find
  10. Federico Fancellu and Bonnie Webber. 2015. Translating Negation: A Manual Error Analysis. In Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExProM 2015), pages 2–11, Denver, Colorado. Association for Computational Linguistics. find
  11. Helmut Schmid, Arne Fitschen, and Ulrich Heid. 2004. A German Computational Morphology Covering Derivation, Composition, and Inflection. In Proceedings of the IVth International Conference on Language Resources and Evaluation (LREC 2004), pages 1263–1266. find
  12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle. Association for Computational Linguistics. find
  13. Rico Sennrich and Barry Haddow. 2015. A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2081–2087, Lisbon, Portugal. Association for Computational Linguistics. find
  14. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016. Neural Machine Translation in Linear Time. ArXiv e-prints. find
  15. Rico Sennrich and Beat Kunz. 2014. Zmorge: A German Morphological Lexicon Extracted from Wiktionary. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik, Iceland. find
  16. Philipp Koehn and Hieu Hoang. 2007. Factored Translation Models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 868–876, Prague, Czech Republic. Association for Computational Linguistics. find
  17. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. Fully Character-Level Neural Machine Translation without Explicit Segmentation. ArXiv eprints, October. find our card
  18. Rico Sennrich, Martin Volk, and Gerold Schneider. 2013. Exploiting Synergies Between Open Resources for German Dependency Parsing, POS-tagging, and Morphological Analysis. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2013, pages 601–609, Hissar, Bulgaria. find
  19. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. ArXiv e-prints, November. find our card
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh Neural Machine Translation Systems for WMT 16. In Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers, pages 368–373, Berlin, Germany, August. Association for Computational Linguistics. find our card
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. find
  22. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pages 3104–3112, Montreal, Quebec, Canada. find
  23. Dominikus Wetzel and Francis Bond. 2012. Enriching Parallel Corpora for Statistical Machine Translation with Semantic Negation Rephrasing. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 20– 29, Jeju, Republic of Korea, July. Association for Computational Linguistics. find
  24. Philip Williams and Philipp Koehn. 2011. Agreement Constraints for Statistical Machine Translation into German. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 217–226, Edinburgh, UK. Association for Computational Linguistics. find
  25. Joern Wuebker, Spence Green, John DeNero, Sasa Hasan, and Minh-Thang Luong. 2016. Models and Inference for Prefix-Constrained Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75. Association for Computational Linguistics. find
  26. Daniel Zeman, Mark Fishel, Jan Berka, and Ondřej Bojar. 2011. Addicter: What is wrong with my translations? The Prague Bulletin of Mathematical Linguistics, 96:79–88. find