Elamur Research⌕
☰
home / papers / 2019 / Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Summary

This paper presents a submission to the WMT19 shared task on filtering noisy web-crawled parallel data for low-resource language pairs. The task supplied very noisy Paracrawl corpora for Nepali–English and Sinhala–English. Participants scored every sentence pair, and the top-ranked pairs (1M and 5M English words) were used to train Moses and fairseq systems, which were evaluated with SacreBLEU on the flores test sets.

The authors built a LASER-style multilingual sentence encoder using only the permitted clean parallel data. They then rated each candidate pair by the similarity of its two embeddings, normalized with a k-nearest-neighbor ratio margin. They compared this approach with Zipporah, Bicleaner and dual conditional cross-entropy, and also combined all the scores through a positive-unlabeled ensemble classifier.

LASER clearly outperformed the other individual scorers, and ensembling added only small further improvements. The submission ranked first on the 1M condition for both language pairs. The results suggest that embedding-based filtering works well when data is scarce and remains reasonable even when the language was never seen in training.

Key points

  • Task data: noisy corpora of 40.6M English tokens (Nepali–English) and 59.6M English tokens (Sinhala–English), evaluated with fairseq NMT and Moses SMT on flores.
  • Encoder design: a single shared five-layer BLSTM (512-dim) with max-pooling, a 50k joint BPE vocabulary built with fastBPE, and an LSTM decoder that always translates into English. Hindi–English data was merged with Nepali–English because the two languages share a script.
  • LASER vs. other scorers: with a local neighborhood, LASER averaged 0.71 BLEU above the best non-LASER method, and 0.94 BLEU on the 1M condition.
  • Ensemble: used PU learning with bagging of 100 RBF-kernel SVMs and two training iterations.
  • Official results: 1M submissions beat the runner-ups by 1.3 BLEU (ne-en) and 1.4 BLEU (si-en), but the 5M results were weaker.
  • Pre-trained LASER (93 languages, no Nepali):
    • On si-en 1M it scored 7.82, ahead of the local model's 7.49.
    • On zero-shot ne-en 1M it scored 6.06, close to Bicleaner's 6.12.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

53 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 45 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5435

@inproceedings{chaudhary2019low,
  title = {{Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings}},
  author = {Chaudhary, Vishrav and Tang, Yuqing and Guzmán, Francisco and Schwenk, Holger and Koehn, Philipp},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)},
  year = {2019},
  doi = {10.18653/v1/w19-5435},
  url = {https://doi.org/10.18653/v1/w19-5435}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Annual Meeting of the Association for Computational Linguistics (ACL), demo session. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations (ICLR). find our card
  3. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L Forcada. 2018. Findings of the WMT 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 726– 739, Belgium, Brussels. Association for Computationa… find our card
  4. Mikel Artetxe and Holger Schwenk. 2018a. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. arXiv preprint arXiv:1811.01136. find our card
  5. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 5039–5049, Belgium, Brussels. Association for Computational Linguistics. find
  6. Mikel Artetxe and Holger Schwenk. 2018b. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. arXiv preprint arXiv:1812.10464. find our card
  7. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. arXiv preprint arXiv:1902.01382. find
  8. Fantine Mordelet and J-P Vert. 2014. A bagging svm to learn from positive and unlabeled examples. Pattern Recognition Letters, 37:201–209. find
  9. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  10. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696, Sofia, Bulgaria. find our card
  11. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, volume 1804.08771, pages 186–191, Belgium, Brussels. Association for Computational Linguistics. find our card
  12. Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 901–908, Belgium, Brussels. Association for Computational Linguistics. find our card
  13. Vı́ctor M Sánchez-Cartagena, Marta Bañón, Sergio Ortiz Rojas, and Gema Ramı́rez. 2018. Prompsit’s submission to WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 955–962, Belgium, Brussels. Association for Comp… find our card
  14. Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74–83, Melbourne, Australia. Association for Computational Linguistics. find our card
  15. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 228– 234, Australia, Melbourne. Association for Computational Linguistics. find
  16. Huda Khayrallah, Hainan Xu, and Philipp Koehn. 2018. The JHU parallel corpus filtering systems for WMT 2018. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 909–912, Belgium, Brussels. Association for Computational Linguistics. find
  17. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2945–2950, Denmark, Cophenhagen. Association for Computational Linguistics. find
  18. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan M. Pino. 2019. Findings of the wmt 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, Volume 2: Shared Task Papers, Florence, Italy. Association for C… find our card
  19. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Ondrej Bojar Chris Dyer, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/chaudhary-2019-low-resource-corpus-filtering-multilingual/