Elamur Research⌕
☰
home / papers / 2018 / Filtering and Mining Parallel Data in a Joint Multilingual Space

Filtering and Mining Parallel Data in a Joint Multilingual Space

Paper · 2018 · Holger Schwenk

Summary

Neural MT systems are sensitive to noisy training bitexts, and much useful parallel text sits unaligned in comparable corpora such as news. Schwenk proposes one method for both cleaning existing bitexts and mining new ones. A single shared encoder (a 3-layer BLSTM) is trained on nine Europarl languages without any language indicator. It maps sentences from any of these languages into a common 1024-dimensional space. The cosine distance between a sentence pair in that space is then thresholded: per pair to filter a corpus, or over all source-target combinations to mine one. No extra classifiers or hand-crafted features are needed.

On the BUCC shared task, the approach reaches F1 scores in the low-to-mid 70s across four language pairs. It trails only the heavily tuned VIC system. Filtering Common Crawl improves a competitive WMT'14 En→De fairseq-py baseline by 0.3 BLEU, and mined news text helps a system trained only on human-translated data.

The approach matters because it is generic: it works for any pair within the joint space, needs only a threshold tuned per task, and does not depend on the MT architecture.

Key points

  • Encoder: a shared BLSTM trained on about 2M Europarl sentences per language for nine languages, with a 20k joint BPE vocabulary and max-pooled sentence vectors. The ru/zh experiments used earlier embeddings trained on the UN corpus.

  • BUCC test F1:

    • en-fr 75.8
    • en-de 76.9
    • en-ru 73.8
    • en-zh 71.6

    These clearly beat the RALI, NLP2CT and H2 systems (except roughly matching H2 on en-fr) but stay below VIC.

  • Filtering Common Crawl:

    • Pre-processing removed about 19% of the corpus: sentences with more than 3 commas, sentences of 50+ words, and pairs failing language ID (about 6% had the wrong language).
    • Distance filtering at threshold 1.0 then raised BLEU from 24.75 to 25.06, a gain of 0.3 BLEU while discarding 25% of the training data.

  • Mining WMT news: the search compared 144M English and 187M German sentences, retrieving k=20 nearest neighbours per source sentence with FAISS.

    • Mined data alone gave 4.18–7.77 BLEU.
    • Adding it to Europarl+News Commentary gave +0.45 BLEU (21.87→22.32).
    • Adding it to the filtered-Common-Crawl system gave no gain.

  • Sentence length: mined sentences average about 8 words, compared with 24 in the WMT'14 training data. Erroneous alignments tend to occur on sentences dominated by named entities or numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Holger Schwenk. 2018. Filtering and Mining Parallel Data in a Joint Multilingual Space. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2037

@inproceedings{schwenk2018filtering,
  title = {{Filtering and Mining Parallel Data in a Joint Multilingual Space}},
  author = {Holger Schwenk},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2037},
  url = {https://doi.org/10.18653/v1/p18-2037}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734. our card
  2. Ashish Vaswani et al. 2017. Attention is all you need. In NIPS. our card
  3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In EMNLP, pages 355–362. our card
  4. Melvin Johnson et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. In https://arxiv.org/ abs/1611.04558. ↗
  5. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2018. Extracting parallel sentences from comparable corpora with STACC variants. In BUCC. find
  6. Chongman Leong, Derek F. Wong, and Lidia S. Chao. 2018. Um-paligner: Neural network-based parallel sentence identification model. In BUCC. find
  7. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and artificial noise both break neural machine translation. In https://arxiv.org/abs/1711. 02173. ↗
  8. Minh-Thang Luong, Hieu Pham, and Christophe D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP, pages 1412–1421. our card
  9. Houda Bouamor and Hassan Sajjad. 2018. H2@bucc18: Parallel sentence extraction from comparable coprora using multlingual sentence embeddings. In BUCC. find
  10. Aditua Mogadala and Achim Rettinger. 2016. Bilingual word embeddings from parallel and non-parallel corpora for cross-language classification. In NAACL, pages 692–702. find
  11. Sarath Chandar, Mitesh M. Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2013. Multilingual deep learning. In NIPS DL wshop. find
  12. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477–504. our card
  13. Junyoung Chunga, Kyunghyun Choa, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In https://arxiv.org/abs/1603.06147. ↗ our card
  14. Hieu Pham, Minh-Thang Luong, and Christopher D. Manning. 2015. Learning distributed representations for multilingual text sequences. In Workshop on Vector Space Modeling for NLP. find
  15. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017. An empirical analysis of nmt-derived interlingual embeddings and their use in parallel sentence identification. IEEE Journal of Selected Topics in Sigmal Processing, pages 1340–1348. find
  16. Sadaf Abdul Rauf and Holger Schwenk. 2009. On the use of comparable corpora to improve SMT performance. In EACL, pages 16–23. find
  17. Philip Resnik and Noah A. Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29:349– 380. our card
  18. Thierry Etchegoyhen and Andoni Azpeitia. 2016. Set-theoretic alignment of comparable corpora. In ACL. find
  19. Lucı́a Santamarı́a and Amittai Axelrod. 2017. Data selection with cluster-based language difference models and cynical selection. In IWSLT, pages 137–145. find
  20. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In ICML. our card
  21. Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. In ACL workshop on Representation Learning for NLP. our card
  22. Francis Grégoire and Philippe Langlais. 2017. find
  23. Bucc 2017 shared task: a first attempt toward a deep learning framework for identifying parallel sentences in comparable corpora. In BUCC, pages 46–50. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In ACL, pages 86–96. find
  25. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… our card
  26. Masao Utiyama and Hitoshi Isahara. 2003. Reliable measures for aligning Japanese-English news articles and sentences. In ACL, pages 72–79. find
  27. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In ACL, pages 58–68. our card
  28. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In EMNLP, pages 1400–1410. our card
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In ACL, pages 1715–1725. our card
  30. Yonghui Wu et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. In https://arxiv.org/abs/1610.05011. ↗
  31. Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Cross-lingual sentiment classification with bilingual document representation learning. In ACL. find
  32. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spottign parallel sentences in comparable corpora. In BUCC, pages 60–67. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwenk-2018-filtering-mining-parallel-data-joint/