Elamur Research⌕
☰
home / papers / 2020 / Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment

Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment

Summary

Web-crawled parallel text is plentiful but very noisy, and that noise hurts neural MT. This matters most for low-resource pairs, where little clean data exists. The third WMT shared task on this problem targeted Pashto–English and Khmer–English. It also added a new subtask: aligning sentences within crawled document pairs. Organizers released noisy corpora drawn from CCAligned and Paracrawl, along with small clean parallel sets, related-language Hindi data, Wikipedia/CommonCrawl monolingual text and pre-trained mBART models. Participants returned one quality score per sentence pair. The organizers used these scores to select subsets totalling a fixed number of English words. They trained MT systems on each subset, both from scratch and by fine-tuning mBART, and measured BLEU on multi-domain Wikipedia test data. Ten teams took part. For filtering, Alibaba and Huawei led the field. Teams that re-aligned the documents themselves outperformed the provided sentence pairs, with Bytedance gaining several BLEU points. The task serves as a reusable benchmark for data cleaning in low-resource settings.

Key points

  • Noisy data sizes (English tokens): about 11.6M words for Pashto–English and 58.3M words for Khmer–English. The official selection target was 5M words, and 2M, 3M and 7M subsets were also analysed.
  • Baseline and top filtering systems: LASER embedding scores were the baseline. Alibaba and Huawei produced the strongest filtering submissions, with NRC, UA-Prompsit and Microsoft close behind.
  • Best alignment system: Bytedance beat the provided pairs scored with LASER in every condition:
    • Pashto from scratch: 7.7 → 11.5 BLEU
    • Pashto with mBART: 10.3 → 12.9 BLEU
    • Khmer from scratch: 8.4 → 12.7 BLEU
    • Khmer with mBART: 12.9 → 15.5 BLEU
  • Subset size for alignment entries: most alignment submissions performed best at 7M words, suggesting they recovered more usable sentence pairs.
  • Differences between submissions: average sentence length in selected Pashto data ranged from roughly 12 words per sentence (NRC) to 29 (AFRL). Leading submissions shared about 60% of their selected pairs, while Alibaba and Huawei overlapped by around 80%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

57 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 51 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.78

@inproceedings{koehn2020findings,
  title = {{Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment}},
  author = {Koehn, Philipp and Chaudhary, Vishrav and El-Kishky, Ahmed and Goyal, Naman and Chen, Peng-Jen and Guzmán, Francisco},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.78},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.78}
}

References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Table 11: Number of sentences and the corresponding average sentence length (counting English words) for Pashto.* Khmer Sentences Words/S Alibaba Huawei JHU Kejriwal 0.8-5 JHU Kejriwal 0.8-5-filt JHU Kejriwal 0.8-6 LASER Microsoft NRC UA-Prompsit 258,044 278,534 218,851 191,864 182,126 240,978 256,7… find our card
  2. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136–144, Portland, Oregon. Association for Computational Li… find
  3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362, Edinburgh, Scotland, UK. Association for Computational Linguistics. find our card
  4. Haluk Açarçiçek, Talha Çolakoğlu, pınar ece aktan hatipoğlu, Chong Hsuan Huang, and Wei Peng. 2020. Filtering noisy parallel corpus using transformers with proxy task learning. In Proceedings of the Fifth Conference on Machine Translation (WMT). *Table 12: Number of sentences and the correspon… find our card
  5. UA-Prompsit NRC alignment NRC Microsoft LASER JHU-Koerner laser-lm JHU-Koerner dual-xent JHU-Kejriwal 0.9-5 JHU-Kejriwal 0.9-0 JHU-Kejriwal 0.8-5 Huawei Edinburgh Bytedance alignment 7.6% 4.6% 62.8% 45.2% 5.6% 3.0% 0.2% 2.0% 0.0% 1.6% 0.0% 0.0% 0.9% 12.7% 42.2% 7.4% Alibaba alignment Unique 172145 3… find

This page in Markdown → https://md.research.elamur.ai/free/papers/koehn-2020-findings-wmt-2020-shared-task/