Elamur Research⌕
☰
home / papers / 2018 / Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora

Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora

Summary

Large web-crawled bitexts such as Paracrawl are cheap and abundant but full of misaligned, untranslated or wrong-language pairs, and neural MT is known to suffer more from such noise than phrase-based systems. The author proposes dual conditional cross-entropy filtering as a way to score every pair in a noisy German–English corpus. Two translation models trained in opposite directions (de→en and en→de) on a 1M-sentence sample of clean WMT data compute word-normalized conditional cross-entropies for each pair. The score rewards pairs where both directions agree and also penalizes pairs that both models consider unlikely. It is combined with langid-based language filtering and a Moore–Lewis-style domain score. The method was tested in the WMT2018 parallel corpus filtering task, where subsets of 10M and 100M words drawn from about 1 billion words of Paracrawl were used to train Moses and Marian systems. NMT trained only on the filtered Paracrawl data outperformed a baseline trained on the full clean WMT18 data. The submission ranked first overall, which shows that selection criteria closely tied to NMT's own training objective are effective.

Key points

  • Scoring formula: The adequacy score combines two terms: the absolute gap between H(y|x) and H(x|y), and the average of the two cross-entropies. It is then mapped into the range (0, 1] via exp(−·). The final score multiplies this with the other partial scores.
  • Language filtering: The language-ID filter (Python langid, requiring German source and English target) discarded close to 70% of candidate pairs. Without it, adequacy filtering performed worse than random selection.
  • Ablation: Removing the disagreement (absolute difference) term lowered BLEU by roughly 1 point. A cosine-similarity score based on a multilingual encoder (W de↔en) did worse than the adq score.
  • Submitted configuration: The submission was de-en·adq·dom^0.25. Development was optimized only for the NMT 100M subtask, using Marian on WMT2016/2017 test sets.
  • Shared task results: Among 48 submissions, it placed first in SMT 100M (26.50), NMT 10M (28.62) and NMT 100M (32.05), and third in SMT 10M (24.45). It was also first in the combined sum (111.63 vs. 110.09). Scores were averaged across six test sets: the 2018 news test set, the 2017 IWSLT set, and the Acquis, EMEA, Global Voices and KDE collections.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

98 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6478

@inproceedings{junczysdowmunt2018dual,
  title = {{Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora}},
  author = {Junczys-Dowmunt, Marcin},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6478},
  url = {https://doi.org/10.18653/v1/w18-6478}
}

References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362. Association for Computational Linguistics. find our card
  2. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. CoRR, abs/1711.02173. find
  3. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pages 1019–1027. find our card
  4. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… find our card
  5. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 820–828. Curran Associates, Inc. find
  6. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, Syste… find our card
  7. Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74–83, Melbourne, Australia. Association for Computational Linguistics. find our card
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL.… find
  9. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics. find our card
  10. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220–224, Uppsala, Sweden. Association for Computational Linguistics. find our card
  11. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228–234. Association for Computational Linguistics. find
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dowmunt-2018-dual-conditional-cross-entropy-filtering/