Elamur Research
☰
home / papers / 2019 / CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1911.04944

Summary

Parallel corpora are scarce, especially for language pairs that do not include English, and existing sources such as parliamentary or UN proceedings cover mostly formal, political topics. The authors scale LASER-based, margin-scored bitext mining from Wikipedia (the WikiMatrix approach) to the web. They use ten snapshots of the curated CCNet version of Common Crawl, which contain 32.7 billion unique sentences in 38 languages. Most bitext miners first align documents and then search for sentence pairs only inside them. This work skips that step and compares every source sentence against every target sentence in a shared multilingual embedding space, which required a redesigned, heavily parallelized pipeline. The resulting resource, CCMatrix, contains about 4.5 billion sentence pairs, many of them directly between non-English languages. The authors trained NMT models only on the mined data, with no human-translated bitext. These models beat the best published single systems on several WMT'19 directions and outperformed WAT'19 submissions for Russian–Japanese. The results show that brute-force global mining over web-scale text can produce high-quality training data across many language pairs.

Key points

  • Scale: About 4.5B parallel sentences were mined, of which 661M are aligned with English. 20 language pairs exceed 30M pairs, and 112 exceed 10M. Only one snapshot was used for English because English web text is so abundant.
  • Method: Pairs are scored with a margin criterion over cosine similarity in the LASER multilingual sentence embedding space, with no document-level preselection. The corpus was built with the CCNet pipeline: deduplication, fastText language ID and Wikipedia-LM perplexity filtering.
  • TED evaluation: Average BLEU is 16.3 over all pairs and 26.9 for pairs involving English. 46 of 702 pairs score above 30 BLEU, compared with 10 of 1620 in WikiMatrix. The best pair is Norwegian→English at 45.2 BLEU.
  • WMT'19: Training used CCMatrix data only, for example 40.6M En-De pairs at margin ≥1.07. En→De reaches 44.7 BLEU on newstest2019, compared with 40.4 for the best single WMT-bitext model and 44.9 for the top ensemble. Adding the WMT bitext raises En-De to 50.9 on newstest2018 and 45.1 on newstest2019.
  • WAT'19 Russian–Japanese: The model was trained on 9.3M direct pairs (margin 1.06) and scored 18.20 BLEU into Japanese, compared with the best WAT'19 result of 16.41. This comparison is not strictly fair, because WAT participants were limited to the resources provided by the organizers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

216 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 204 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.04944 · arXiv 1911.04944

@misc{schwenk2019ccmatrix,
  title = {{CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB}},
  author = {Schwenk, Holger and Wenzek, Guillaume and Edunov, Sergey and Grave, Edouard and Joulin, Armand},
  year = {2019},
  doi = {10.48550/arxiv.1911.04944},
  eprint = {1911.04944},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.04944}
}

References

51 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the Use of Comparable Corpora to Improve SMT performance. In EACL, pages 16–23. find
  2. Sisay Fissaha Adafre and Maarten de Rijke. 2006. Finding similar sentences across multiple languages in Wikipedia. In Proceedings of the Workshop on NEW TEXT Wikis and blogs and other dynamic text sources. find
  3. Ahmad Aghaebrahimian. 2018. Deep neural networks at the service of multilingual parallel sentence extraction. In Coling. find
  4. Mikel Artetxe and Holger Schwenk. 2018a. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. https://arxiv.org/abs/1811.01136. ↗ our card
  5. Mikel Artetxe and Holger Schwenk. 2018b. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. In https: //arxiv.org/abs/1812.10464. find our card
  6. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2017. Weighted Set-Theoretic Alignment of Comparable Sentences. In BUCC, pages 41–45. find
  7. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2018. Extracting Parallel Sentences from Comparable Corpora with STACC Variants. In BUCC. find
  8. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translatio… find our card
  9. Houda Bouamor and Hassan Sajjad. 2018. H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings. In BUCC. find
  10. Christian Buck and Philipp Koehn. 2016. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 554–563, Berlin, Germany. Association for Computational Linguistics. find
  11. Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  12. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017. An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification. IEEE Journal of Selected Topics in Signal Processing, pages 1340–1348. find
  13. Miquel Esplà-Gomis and Mikel L. Forcada. 2010. Combining content-based and url-based heuristics to harvest aligned bitexts from multilingual sites with bitextor. The Prague Bulletin of Mathematical Linguistics, 9:77–86. find
  14. Thierry Etchegoyhen and Andoni Azpeitia. 2016. Set-Theoretic Alignment for Comparable Corpora. In ACL, pages 2009–2018. find
  15. Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with structured dropout. find our card
  16. Simon Gottschalk and Elena Demidova. 2017. Mul-tiwiki: Interlingual text passage alignment in Wikipedia. ACM Transactions on the Web (TWEB), 11(1):6. find
  17. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. https://arxiv.org/abs/1802.06893. ↗ our card
  18. Francis Grégoire and Philippe Langlais. 2017. find
  19. BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora. In BUCC, pages 46– 50. find
  20. Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective Parallel Corpus Mining using Bilingual Sentence Embeddings. arXiv:1807.11906. find
  21. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… find our card
  22. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. arXiv preprint arXiv:1702.08734. find our card
  23. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. https://arxiv.org/abs/1607.01759. ↗
  24. H. Jégou, M. Douze, and C. Schmid. 2011. Product quantization for nearest neighbor search. IEEE Trans. PAMI, 33(1):117–128. find
  25. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit. find
  26. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan M. Pino. 2019. Findings of the wmt 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, Volume 2: Shared Task Papers, Florence, Italy. Association for C… find
  27. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726–739, Belgium, Brussels. Association for Computational Linguist… find
  28. P. Lison and J. Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In LREC. find our card
  29. Mehdi Zadeh Mohammadi and Nasser GhasemAghaee. 2010. Building bilingual parallel corpora based on Wikipedia. In 2010 Second International Conference on Computer Engineering and Applications, pages 264–268. find
  30. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Computational Linguistics, 31(4):477–504. find our card
  31. Toshiaki Nakazawa, Nobushige Doi, Shohei Higashiyama, Chenchen Ding, Raj Dabre, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, and Sadao Kurohashi. 2019. Overview of the 6th workshop on Asian translation. In Proceedings of the 6th Workshop on Asian Translat… find
  32. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Florence, Italy. Association for Computat… find our card
  33. P Otero, I López, S Cilenis, and Santiago de Compostela. 2011. Measuring comparability of multilingual corpora extracted from Wikipedia. Iberian Cross-Language Natural Language Processings Tasks (ICL), page 8. find
  34. Pablo Gamallo Otero and Isaac González López. 2010. Wikipedia as multilingual source of comparable corpora. In Proceedings of the 3rd Workshop on Building and Using Comparable Corpora, LREC, pages 21–25. find
  35. Alexandre Patry and Philippe Langlais. 2011. Identifying parallel documents from a large bilingual collection of texts: Application to parallel article extraction in Wikipedia. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 87– 95.… find
  36. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  37. Philip Resnik. 1999. Mining the Web for Bilingual Text. In ACL. find
  38. Philip Resnik and Noah A. Smith. 2003. The Web as a Parallel Corpus. Computational Linguistics, 29(3):349–380. find our card
  39. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In ACL, pages 228–234. find
  40. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. In http://arxiv. org/abs/1907.05791. find our card
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find
  42. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In NAACL, pages 403–411. find
  43. Meng Sun, Bojian Jiang, Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2019. Baidu neural machine translation systems for wmt19. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 374–381, Florence, Italy. Association for Computational Ling… find
  44. J. Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In LREC. find
  45. Chen-Tse Tsai and Dan Roth. 2016. Cross-lingual wikification using multilingual embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 589–598. find
  46. Dan Tufis, Radu Ion, S , tefan Daniel, Dumitrescu, and Dan S , tefănescu. 2013. Wikipedia as an smt training corpus. In RANLP, pages 702–709. find
  47. Masao Utiyama and Hitoshi Isahara. 2003. Reliable Measures for Aligning Japanese-English News Articles and Sentences. In ACL. find
  48. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. CCNet: extracting high quality monolingual datasets from web crawl data. https://arxiv.org/abs/1911.00359. ↗ our card
  49. Yinfei Yang, Gustavo Hernández Ábrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In https://arxiv.org/abs/1902.08564. — [arXiv:1902.08564](https://arxiv.org/abs/1902.08564) ↗
  50. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In LREC. find our card
  51. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the Third BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora. In Proceedings of the 11th Workshop on Building and Using Comparable Corpora. find