Elamur Research
☰
home / papers / 2020 / The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT

The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.wmt-1.139.pdf

Summary

Research on low-resource and zero-shot MT often fakes scarcity by removing one language pair from multi-parallel, high-resource corpora such as Europarl. This probably overstates how well transfer learning works in practice. Tiedemann responds with the Tatoeba Translation Challenge, a benchmark that reflects the uneven and noisy data that most languages actually have.

  • Training data: the entire OPUS collection, cleaned of encoding errors, escaped characters and mislabelled text (using CLD2). It is shuffled, but each sentence keeps a tag for its source corpus.
  • Dev and test data: taken from crowd-sourced Tatoeba translations. Tatoeba and WMT test sentences are kept out of training.
  • Labels: every example has an ISO-639-3 code and an automatically detected ISO-15924 script tag.
  • Monolingual data: Wikimedia text is included for back-translation or unsupervised work.

Baseline models show that family-level multilingual systems help most in zero- and few-shot settings, but scores there stay very low. Bilingual models trained on the data come close to the best WMT 2019 English–German results. The release substantially widens open MT evaluation beyond the narrow language coverage of existing benchmarks.

Key points

  • Scale: over 500GB compressed data, 2,961 language pairs and 555 languages. English–French is the largest pair with more than 180 million sentence pairs. 173 pairs exceed 10 million examples, and 298 pairs count as high-resource (at least 1 million).
  • Monolingual data: drawn from Wikipedia, Wikibooks, Wikinews, Wikiquote and Wikisource. Sentences are split with UDPipe (models trained on UD treebanks v2.4) and the Moses splitter, and document order is preserved.
  • Zero-shot (Awadhi–English, no training data): the ine-eng model reached 10.0 BLEU. A naive all-languages tatoeba-zero model scored only 0.1.
  • Few-shot (English–Faroese, 3,613 training pairs): eng-gem performed best at 9.4 BLEU and chr-F2 0.318.
  • WMT 2019 news, no domain tuning: eng-deu scored 42.4 BLEU (best submission 44.9) and deu-eng scored 40.5 (best 42.8). Broader multilingual models dropped steadily in quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 142 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.139

@inproceedings{tiedemann2020tatoeba,
  title = {{The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT}},
  author = {Tiedemann, Jörg},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.139},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.139}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), pages 261–268, Trento, Italy. find
  2. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875… find our card
  3. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. find our card
  4. Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas. Associa… find
  5. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-u files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4290– 4297… find
  6. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. find our card
  7. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of LREC, Istanbul, Turkey. find
  8. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association of Computational L… find
  9. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  10. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, Syste… find our card
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. find
  12. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. Citeseer. find
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  14. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… find our card
  15. Surafel M. Lakew, Marcello Federico, Matteo Negri, and Marco Turchi. 2018. Multilingual neural machine translation for low-resource languages. IJCoL - Italian Journal of Computational Linguistics, 4(1). Emerging Topics at the Fourth Italian Conference on Computational Linguistics. find