📄 Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions

Summary

Methods that filter noisy web-crawled bitext work well for high-resource languages, but it was unclear how they behave when very little clean data exists to train the scoring models. The WMT 2019 shared task tested this for Nepali–English and Sinhala–English. Organizers released noisy corpora of roughly 50–60 million English words each, built with the Paracrawl pipeline, along with clean bitext amounting to merely several million words. Participants assigned a quality score to every sentence pair. The top-ranked pairs were then cut to subsets of 1 million and 5 million English words. Each subset was used to train phrase-based statistical and neural MT systems, which were evaluated with BLEU on the flores Wikipedia test sets.

Eleven teams took part, and their results varied widely. Neural systems peaked on fairly small, clean subsets. Statistical systems kept improving as more data was added, suggesting they are more tolerant of noise. The task offers a benchmark for low-resource data cleaning.

Key points

• The setup is far smaller than WMT 2018:
  • About 50–60M noisy words per language pair, compared with 1 billion for German–English.
  • Only a few million words of clean data, compared with more than 100M.
• Submitted scores were judged by training SMT and NMT systems on the 1M- and 5M-word subsets (about 2% and 10% of the data) and measuring BLEU on the flores test sets.
• Extra subset sizes were tested for 7 systems. NMT optima ranged from 0.7M to 3M words: around 1M for Nepali and around 2M for Sinhala.
• SMT improved steadily with more data and beat NMT at 5M words. The best quality came from roughly 2–6% of the raw corpus, versus 10% or more in the 2018 German–English task.
• Submissions differed sharply in sentence-length preference. For example, Nepali 1M-word subsets averaged anywhere from about 7 to 35 English words per sentence.