Elamur Research
☰
home / papers / 2022 / NTREX-128 – News Test References for MT Evaluation of 128 Languages

NTREX-128 – News Test References for MT Evaluation of 128 Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.sumeval-1.4.pdf

Summary

Evaluating massively multilingual MT models requires test data, but building test sets for over a hundred languages is expensive, so only a few such benchmarks exist (TICO-19, FLORES-101, FLORES-200). Microsoft researchers address this gap with NTREX-128. Professional translators rendered the English side of the WMT19 news test set (newstest2019) into 128 languages. Each translator had to be a native speaker of the target language with a strong command of English. Post-editing of machine output was banned to avoid reference bias. An independent vendor then rated every delivered segment with source-based direct assessment (src-DA) in the Appraise framework, and weak segments went back to the translation provider for repair. A COMET-src experiment found that, for many languages, translated (translationese) input scores higher than authentic source text. Scores also looked unreliable for languages absent from COMET's training data. The authors therefore recommend using NTREX-128 only for English→X evaluation. The release adds a SacreBLEU-compatible benchmark for multilingual MT research.

Key points

  • NTREX-128 consists of 123 documents (1,997 sentences, about 42k words) drawn from WMT19 newstest2019, with English as the source and 128 languages on the target side.
  • Requirements for references: native-speaker translators bilingual in English, and no post-edited MT output.
  • Quality control used src-DA scores on a 0–100 scale. Segments below 25 were treated as defective and those in [25, 50) as suspect. Everything under 50 was returned for fixing, and all segments were checked rather than a random sample.
  • In the COMET-src experiments, scores were not comparable across language pairs. Reverse-direction (translationese) input often scored higher. Some languages unseen by COMET produced broken-looking scores.
  • For comparison, TICO-19 covers 37 languages, FLORES-101 covers 101, and FLORES-200 covers 200, with differing domains and language sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

86 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.sumeval-1.4

@inproceedings{federmann2022ntrex,
  title = {{NTREX-128 – News Test References for MT Evaluation of 128 Languages}},
  author = {Federmann, Christian and Kocmi, Tom and Xin, Ying},
  booktitle = {Proceedings of the First Workshop on Scaling Up Multilingual Evaluation},
  year = {2022},
  doi = {10.18653/v1/2022.sumeval-1.4},
  url = {https://doi.org/10.18653/v1/2022.sumeval-1.4}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO-19: the Translation initiative for COvid-19. arXiv:2007.01788. ↗
  2. James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Janice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Francisco Guzmán Philipp Koehn Alexandre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk Jeff Wang #NLLB Team, Marta R. Costa-jussà. 2022. No language left behind: Scaling human-centered machine translation. find
  3. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. find our card
  4. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  6. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  7. A Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation. License See the LICENSE file in our repository. In addition to these license terms we ask that you cite this paper when using NTREX-128 in your work. Thank you. find our card