Elamur Research⌕
☰
home / papers / 2018 / Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task

Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task

Summary

The paper describes Prompsit's submission to the corpus-filtering shared task organised at WMT 2018, which targets parallel data. Participants start from a very noisy German–English corpus of roughly 1 billion words crawled from the web. They must select subsets of 10M and 100M words, and each subset is judged by the BLEU score of SMT and NMT systems trained on it, using test sets from several domains. Prompsit's pipeline has three stages:

  • Rule filtering: hand-written rules from the Bicleaner tool discard clearly broken pairs, such as wrong language, encoding errors or mismatched lengths.
  • Translation classifier: a Random Forest estimates how likely each remaining pair is a genuine translation. It combines bilingual-dictionary similarity features with surface features.
  • Ranking: four strategies favour fluent sentences and varied vocabulary. These are n-gram saturation with placeholders, an active-learning-inspired selection, and two language-model perplexity variants.

Every submission clearly beat random selection. On the 100M track the systems ranked close to the top participants, while on the 10M track they landed mid-table. The results suggest that weighting fluency too heavily can shrink vocabulary coverage when the selected corpus is small.

Key points

  • Bicleaner's hard rules kept 22 229 462 of 104 002 521 sentence pairs (21%) before classification.
  • The Random Forest classifier (Scikit-learn, 200 trees, depth 2) used dictionaries built with MGIZA++ alignments. It was trained on 50 000 pairs from news-commentary-v13, with negative examples created by shuffling the English side, and reached 0.98 accuracy at a 0.5 threshold.
  • The four submissions were prompsit-sat, prompsit-al, prompsit-lm and prompsit-lm-nota. N-gram saturation alone left 10 100 275 pairs.
  • On NMT with the 100M dataset, prompsit-lm-nota scored 29.85 BLEU, compared with 29.32 for the classifier alone and 7.66 for random selection. On SMT 100M, prompsit-al was best with 24.46.
  • On the 10M dataset, prompsit-sat was strongest (NMT 22.82, SMT 20.77), while the perplexity-based variants dropped to about 18.5 NMT BLEU and showed more unknown words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

51 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 44 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6488

@inproceedings{sanchezcartagena2018prompsit,
  title = {{Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task}},
  author = {Sánchez-Cartagena, Víctor M. and Bañón, Marta and Ortiz-Rojas, Sergio and Ramírez, Gema},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6488},
  url = {https://doi.org/10.18653/v1/w18-6488}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the use of comparable corpora to improve smt performance. In Proceedings of the 12th Conference of the The difference between these two numbers is the amount of sentences removed by the n-gram saturation algorithm. find
  2. System random only-classifier prompsit-sat prompsit-al prompsit-lm prompsit-lm-nota de 10M 904K 561K 585K 403K 359K 364K en 10M 789K 382K 365K 228K 99K 103K de 100M 3 810K 2 197K 2 246K 2 329K 2 022K 1 969K en 100M 3 364K 1 274K 1 174K 1 162K 910K 879K *Table 2: Vocabulary sizes, expressed in thousa… find
  3. Sauleh Eetemadi, William Lewis, Kristina Toutanova, and Hayder Radha. 2015. Survey of data-selection methods in statistical machine translation. Machine Translation, 29(3):189–223. find
  4. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 355–362, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  5. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Joseph van Genabith. 2017. An empirical analysis of nmt-derived interlingual embeddings and their use in parallel sentence identification. IEEE Journal of Selected Topics in Signal Processing, 11(8):1340–1350. find
  6. Eduard Barbu, Carla Parra Escartín, Luisa Bentivogli, Matteo Negri, Marco Turchi, Constantin Orasan, and Marcello Federico. 2016. The first automatic translation memory cleaning shared task. Machine Translation, 30(3):145–166. find
  7. Gholamreza Haffari, Maxim Roy, and Anoop Sarkar. 2009. Active learning for statistical phrase-based machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL ’09, pages 415–423,… find
  8. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. CoRR, abs/1711.02173. find
  9. Leo Breiman. 2001. Random forests. Machine Learning, 45(1):5–32. find
  10. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 427–436, Stroudsburg, PA, USA. As… find
  11. Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, United Kingdom. find our card
  12. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696, Sofia, Bulgaria. find our card
  13. Chris Dyer. 2009. Using a maximum entropy model to build segmentation lattices for mt. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckerman… find
  14. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics. find our card
  15. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Comput. Linguist., 31(4):477–504. find our card
  16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715–1725. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cartagena-2018-prompsit-s-submission-wmt-2018/