Elamur Research
☰
home / papers / 2017 / A Challenge Set Approach to Evaluating Machine Translation

A Challenge Set Approach to Evaluating Machine Translation

What it is about

The paper proposes challenge sets: small, hand-written sets of English to French sentences, each built to test one structural divergence between the two languages, for example morphology that exposes agreement, stranded prepositions, double-object verbs, subcategorisation frames, idioms and WH-movement. Each item has a yes or no question for a human judge. The authors compare phrase-based and neural systems of 2017. Neural systems beat phrase-based ones on almost every category, idioms being the exception. Some rules are learned by the neural system in simple cases and still fail in longer sentences, which the authors report as brittle generalisation. The method is manual, the sets are small, and native speakers are needed to repeat it.

Ideas

For Mova

The places are tools/translate test data, tools/macula for the source trees and en/src/morph.rs for the forms. The items are small and can live in the repo as data. The set is a dev instrument; any score on it must be read against the held-out rule and the two-run check.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-1263.pdf

Summary

As neural machine translation (NMT) approaches human quality on well-resourced pairs like English–French, surface-overlap metrics such as BLEU become less informative. NMT systems are also opaque, so it is hard to tell which linguistic problems they have actually solved. Isabelle, Cherry and Foster address this with a challenge set: a small collection of short English sentences, each built by hand using linguistic expertise to test one structural divergence between English and French. Human raters judge the system output through simple yes/no questions about the targeted phenomenon. The authors use this set to compare in-house phrase-based (PBMT) and neural systems with Google's GNMT.

Neural systems came out ahead wherever a clear difference appeared, with idioms as the only exception. This advantage is linked to NMT escaping the limits of n-gram language models. Several phenomena still defeated every system. The approach complements random test sets and gives developers a sharper, fine-grained diagnostic signal.

Key points

  • Divergences are grouped into three types: morpho-syntactic, lexico-syntactic and purely syntactic.
    • Morpho-syntactic examples are subject–verb agreement and the subjunctive. French verbs have over 30 inflected forms, while English verbs have about 4–5.
    • Lexico-syntactic examples are argument switching and crossing-movement verbs.
    • Syntactic examples are clitic pronouns and stranded prepositions.
  • To exclude data sparsity, source and reference vocabulary was restricted to words seen at least 100 times in the training corpus. The two deliberate exceptions are "spilt" and the nonce verb "guitared".
  • Three native-speaker annotators rated four MT systems. The annotated challenge set is released in formatted and machine-readable versions.
  • NMT handled agreement, double-object verbs, overlapping subcategorization frames and stranded prepositions (WH-movement) better than PBMT. PBMT did better on idioms.
  • No system solved the following:
    • subject-control verbs;
    • argument switching ("miss" → "manquer à");
    • crossing-movement restructuring ("swim across" → "traverser à la nage");
    • English generic passives that should become French middle-voice constructions.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1263

@inproceedings{isabelle2017challenge,
  title = {{A Challenge Set Approach to Evaluating Machine Translation}},
  author = {Isabelle, Pierre and Cherry, Colin and Foster, George},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1263},
  url = {https://doi.org/10.18653/v1/d17-1263}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the Third International Conference on Learning Representations (ICLR). San Diego, USA. http://arxiv.org/abs/1409.0473. ↗ our card
  2. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 257–267. https://aclweb.org/anthology/D16-1025. ↗
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 131–198. http://www.aclweb.org/anthology/W16-2301. ↗ our card
  4. Colin Cherry. 2013. Improved reordering for phrase-based translation using sparse features. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Atlanta, Georgia, pages 22–31. http://www.aclweb.org/anthology/N13-1003. ↗
  5. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Montréal, Canada, pages 427–436. http://www.aclweb.org/anthology/N12-1047. ↗
  6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1724–1734. http://www.aclweb.org/anthology/D14-1179. ↗
  7. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 1370–1380. http://www.aclweb.org/anthology/P14-1129. ↗
  8. Bonnie J. Dorr. 1994. Machine translation divergences: a formal description and proposed solution. Computational Linguistics 20:4. http://aclweb.org/anthology/J/J94/J94-4004.pdf. ↗ our card
  9. Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and Stephan Vogel. 2016. QCRI machine translation systems for IWSLT 16. In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT). Seattle, Washington. https://workshop2016.iwslt.org/downloads/qcri-machine-translation.pdf. ↗
  10. Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 848–856. http://www.aclweb.org/anthology/D08-1089. ↗
  11. Liang Huang and David Chiang. 2007. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 144–151. http://www.aclweb.org/anthology/P07-1019. ↗
  12. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016a. Is neural machine translation ready for deployment? a case study on 30 translation directions. In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT). Seattle, Washington. find
  13. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Rico Sennrich. 2016b. The amu-uedin submission to the wmt16 news translation task: Attention-based nmt models as feature functions in phrase-based smt. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 319–325. http://www.aclweb.org/anthology/W16-2316. ↗
  14. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. http://www.aclweb.org/anthology/D13-1176. ↗
  15. Margaret King and Kirsten Falkedal. 1990. Using test suites in evaluation of machine translation systems. In Proceedings of the 1990 Conference on Computational Linguistics. Association for Computational Linguistics, Helsinki, Finland. http://aclweb.org/anthology/C/C90/C90-2037.pdf. ↗ our card
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages 311–318. https://doi.org/10.3115/1073083.1073135. ↗ our card