Elamur Research⌕
☰
home / papers / 2018 / Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering

Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering

Summary

The WMT 2018 parallel corpus filtering task asks how to pick out usable sentence pairs from extremely noisy web-crawled bitext. This matters most for neural MT, which is especially sensitive to bad training data. The organizers released a German–English corpus of about 1 billion English tokens from the Paracrawl project. Each line held a sentence pair and its Hunalign score, and URL sources were provided separately. Participants scored every pair for quality. The organizers then used those scores to build 10M-word and 100M-word subsets, trained Moses (phrase-based SMT) and Marian (NMT) systems on them, and measured BLEU on six test sets. Four of those test sets were held back from participants. Clean, general quality was the goal, not domain fit. Seventeen organizations took part. Most systems followed the same pattern: rule-based pre-filtering, then scoring with language models, translation models or lexical probabilities, sometimes combined by a classifier. For NMT, quality peaked at roughly 100M selected words, while SMT was much less sensitive to subset size. The task provides a shared benchmark and reusable tooling for cleaning crawled corpora.

Key points

  • The raw Paracrawl-derived corpus is very noisy:
    • Only about 23% of pairs were judged okay.
    • 41% were misaligned.
    • The rest included untranslated text, wrong or third languages, very short segments and non-linguistic characters.
  • Six evaluation test sets were used: NEWSTEST 2018, IWSLT 2017, ACQUIS, EMEA, GLOBALVOICES and KDE. The last four were drawn from OPUS, and the undisclosed sets were given more weight.
  • Seventeen participants (9 companies, 3 national labs, 5 universities) submitted up to 5 score sets each, 44 in total.
  • Common filtering ingredients included:
    • mismatch checks for numbers, URLs and names;
    • copy detection;
    • language ID;
    • n-gram or neural LMs, NMT scores and bag-of-words lexical translation probabilities;
    • classifiers trained on clean versus synthetic noisy pairs.
  • In additional size experiments, NMT BLEU peaked near 100M words. Several SMT systems stayed within about 1 BLEU across 30–200M words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

82 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6453

@inproceedings{koehn2018findings,
  title = {{Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering}},
  author = {Koehn, Philipp and Khayrallah, Huda and Heafield, Kenneth and Forcada, Mikel L.},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6453},
  url = {https://doi.org/10.18653/v1/w18-6453}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonova, A. and Misyurev, A. (2011). Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136–144, Portland, Oregon. Association for Computational Linguistic… find
  2. Fomicheva, M. and González-Rubio, J. (2018). Maje submission to the wmt2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  3. Hangya, V. and Fraser, A. (2018). An unsupervised system for parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  4. Ash, T., Francis, R., and Williams, W. (2018). The speechmatics parallel corpus filtering system for wmt18. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  5. Junczys-Dowmunt, M. (2018). Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find our card
  6. Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362, Edinburgh, Scotland, UK. Association for Computational Linguistics. find our card
  7. Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Aji, A. F., Bogoychev, N., Martins, A. F. T., and Birch, A. (2018). Marian: Fast neural machine translation in c++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121. As… find our card
  8. Azpeitia, A., Etchegoyhen, T., and Martı́nez garcia, E. (2018). Stacc, oov density and n-gram saturation: Vicomtech’s participation in the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computationa… find
  9. Khayrallah, H. and Koehn, P. (2018). On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74–83. Association for Computational Linguistics. find our card
  10. Barbu, E. and Barbu Mititelu, V. (2018). A hybrid pipeline of rules and machine learning to filter web-crawled parallel corpora. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  11. Khayrallah, H., Xu, H., and Koehn, P. (2018). The jhu parallel corpus filtering systems for wmt 2018. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  12. Belinkov, Y. and Bisk, Y. (2017). Synthetic and natural noise both break neural machine translation. CoRR, abs/1711.02173. find
  13. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), Phuket, Thailand. find
  14. Buck, C. and Koehn, P. (2016). Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 554–563, Berlin, Germany. Association for Computational Linguistics. find
  15. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C. J., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the A… find
  16. Carpuat, M., Vyas, Y., and Niu, X. (2017). Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69–79, Vancouver. Association for Computational Linguistics. find
  17. Cui, L., Zhang, D., Liu, S., Li, M., and Zhou, M. (2013). Bilingual data cleaning for SMT using graph-based random walk. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 340–345, Sofia, Bulgaria. Association for Computational… find
  18. Littell, P., Larkin, S., Stewart, D., Simard, M., Goutte, C., and Lo, C.-k. (2018). Measuring sentence parallelism using mahalanobis distances: The nrc unsupervised submissions to the wmt18 parallel corpus Skadiņš, R., Tiedemann, J., Rozis, R., and Deksne, D. (2014). Billions of parallel words for… find
  19. Lo, C.-k., Simard, M., Stewart, D., Larkin, S., Goutte, C., and Littell, P. (2018). Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The nrc supervised submissions to the parallel corpus filtering task. In Proceedings of t… find
  20. Sánchez-Cartagena, V. M., Bañón, M., Ortiz Rojas, S., and Ramı́rez, G. (2018). Prompsit’s submission to wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find our card
  21. Lu, J., Lv, X., Shi, Y., and Chen, B. (2018). Alibaba submission to the wmt18 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  22. Täger, W. (2011). The sentence-aligned european patent corpus. In Forcada, M. L., Depraetere, H., and Vandeghinste, V., editors, Proceedings of the 15th International Conference of the European Association for Machine Translation (EAMT), pages 177–184. find
  23. Paetzold, G. (2018). Utfpr at wmt 2018: Minimalistic supervised corpora filtering for machine translation. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  24. Taghipour, K., Khadivi, S., and Xu, J. (2011). Parallel corpus refinement as an outlier detection algorithm. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII), pages 414–421. International Association for Machine Translation. find
  25. Papavassiliou, V., Sofianopoulos, S., Prokopidis, P., and Piperidis, S. (2018). The ilsp/arc submission to the wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  26. Tiedemann, J. (2012). Parallel data, tools and interfaces in opus. In Chair), N. C. C., Choukri, K., Declerck, T., Dogan, M. U., Maegaard, B., Mariani, J., Odijk, J., and Piperidis, S., editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbu… find
  27. Pham, M. Q., Crego, J., and Senellart, J. (2018). Systran participation to the wmt2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. van der Wees, M., Bisazza, A., and Monz, C. (2… find
  28. Pinnis, M. (2018). Tilde’s parallel corpus filtering methods for wmt 2018. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  29. Rafalovitch, A. and Dale, R. (2009). United Nations General Assembly resolutions: A six-language parallel corpus. In Proceedings of the Twelfth Machine Translation Summit (MT Summit XII). International Association for Machine Translation. find
  30. Varga, D., Halaácsy, P., Kornai, A., Nagy, V., Németh, L., and Trón, V. (2005). Parallel corpora for medium density languages. In Proceedings of the RANLP 2005 Conference, pages 590–596. find
  31. Rarrick, S., Quirk, C., and Lewis, W. (2011). MT detection in web-scraped parallel corpora. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII), pages 422–430. International Association for Machine Translation. find
  32. Resnik, P. (1999). Mining the web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association of Computational Linguistics (ACL). find
  33. Venugopal, A., Uszkoreit, J., Talbot, D., Och, F., and Ganitkevitch, J. (2011). Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1363–1372, Edinbu… find
  34. Rossenbach, N., Rosendahl, J., Kim, Y., Graça, M., Gokrani, A., and Ney, H. (2018). The rwth aachen university filtering system for the wmt 2018 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistic… find
  35. Wang, R., Marie, B., Utiyama, M., and Sumita, E. (2018). Nict’s corpus filtering systems for the wmt18 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  36. Xu, H. and Koehn, P. (2017). Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2935–2940. Association for Computational Linguistics. find
  37. Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2015). The united nations parallel corpus v1.0. In International Conference on Language Resources and Evaluation (LREC). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/koehn-2018-findings-wmt-2018-shared-task/