Elamur Research⌕
☰
home / papers / 2017 / Findings of the WMT 2017 Biomedical Translation Shared Task

Findings of the WMT 2017 Biomedical Translation Shared Task

Summary

This second edition of the WMT biomedical translation shared task looks at how well machine translation systems handle health and life-science texts between English and other European languages. Such translation can help clinicians and the public read medical content in their own language and help researchers write in English. The task grew from three to ten languages: cs, de, en, fr, hu, pl, pt, es, ro and sv. Systems translated from English into every other language, and from French, Spanish and Portuguese into English.

The organizers built new test sets from four sources:

  • Scielo and EDP Sciences: titles and abstracts of scientific articles.
  • Cochrane and NHS: health documents translated by experts, supplied through the KConnect and HimL projects.

Seven teams sent 82 runs, using both phrase-based SMT and neural MT, often with domain adaptation. Runs were scored with BLEU and judged manually by native speakers. Performance and translation quality rose compared with 2016. Manual judgments mostly matched BLEU rankings, which suggests automatic scores are informative for this domain.

Key points

  • Test sets: Cochrane (25 documents / 467 sentences) and NHS (25 documents / 1044 sentences) each cover eight target languages. Scielo serves the es/pt–en pairs, and EDP replaces it for en/fr.
  • Alignment quality: about 88% of Scielo sentences (aligned with GMA) and 94% of EDP sentences (aligned with YASA) were correct on manually checked samples.
  • Teams: Hunter, kyoto, Lilt, LMU, PJIIT, uedin-nmt and UHH, from five countries. Each team could submit up to three runs per pair and test set, within 10 days.
  • Notable BLEU results: on Scielo, UHH clearly beat the simple baseline. On EDP en2fr, only kyoto outperformed the stronger LIMSI Moses baseline.
  • Limitations and errors:
    • Test sets are small, no significance tests were run, and each comparison was judged by a single expert.
    • Manual review found mistranslated technical terms; one example rendered "vasopressor" as its antonym in Romanian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 30 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4719

@inproceedings{yepes2017findings,
  title = {{Findings of the WMT 2017 Biomedical Translation Shared Task}},
  author = {Yepes, Antonio Jimeno and Névéol, Aurélie and Neves, Mariana and Verspoor, Karin and Bojar, Ondřej and Boyer, Arthur and Grozea, Cristian and Haddow, Barry and Kittner, Madeleine and Lichtblau, Yvonne and Pecina, Pavel and Roller, Roland and Rosa, Rudolf and Siu, Amy and Thomas, Philippe and Trescher, Saskia},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4719},
  url = {https://doi.org/10.18653/v1/w17-4719}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… find our card
  2. Conference on Machine Translation (WMT16) at the Conference of the Association of Computational Linguistics, pages 131–198. Mariana Neves, Antonio Jimeno Yepes, and Aurélie Névéol. 2016. The Scielo Corpus: a Parallel Corpus of Scientific Publications for Biomedicine. In Nicoletta Calzolari (Confe… find
  3. Fabien Cromieres. 2016. Kyoto-NMT: a Neural Machine Translation implementation in Chainer. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations. find
  4. The COLING 2016 Organizing Committee, Osaka, Japan, pages 307–311. find
  5. Fabien Cromieres, Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2016. Kyoto University Participation to WAT 2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016). find
  6. The COLING 2016 Organizing Committee, Osaka, Japan, pages 166–174. find
  7. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The University of Edinburgh’s Neural MT Systems for WMT17. In Proceedings of the Second Conference on Machine Translation (WMT17) at the Conference o… find our card
  8. Mirela-Stefania Duma and Wolfgang Menzel. 2017. Automatic Threshold Detection for Data Selection in Machine Translation. In Proceedings of the Second Conference on Machine Translation (WMT17) at the Conference on Empirical Methods in Natural Language Processing. find
  9. Krzysztof Wolk and Krzysztof Marasek. 2017. PJIIT’s systems for WMT 2017 Conference. In Proceedings of the Second Conference on Machine Translation (WMT17) at the Conference on Empirical Methods in Natural Language Processing. find
  10. Christian Federmann. 2010. Appraise: An open-source toolkit for manual phrase-based evaluation of translations. In In LREC. find
  11. Matthias Huck and Alexander Fraser. 2017. Lmu Munich’s Neural Machine Translation Systems for News Articles and Health Information Texts. In Proceedings of the Second Conference on Machine Translation (WMT17) at the Conference on Empirical Methods in Natural Language Processing. find
  12. Jia Xu, Yi Zong Kuang, Shondell Baijoo, Jacob Lee, Mir Ahmed, Uman Shahzad, Meredith Lancaster, and Chris Carlan. 2017. Supervised Study for Young Machine Translators: Hunter Machine Translation Systems for WMT17. In Proceedings of the Second Conference on Machine Translation (WMT17) at the Conferen… find
  13. Julia Ive, Aurélien Max, and François Yvon. 2016a. Limsi’s contribution to the wmt’16 biomedical translation task. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 469–476. http://www.aclweb.org/anthology/W/W16/W16-233… ↗
  14. Julia Ive, Aurélien Max, François Yvon, and Philippe Ravaud. 2016b. Diagnosing high-quality statistical machine translation using traces of post-edition operations. In Proceedings of the LREC 2016 Workshop: Translation evaluation From fragmented tools and data sets to an integrated ecosystem. Euro… find
  15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yepes-2017-findings-wmt-2017-biomedical-translation/