Elamur Research
☰
home / papers / 2019 / WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1907.05791

Summary

Parallel text is scarce for most language pairs, especially when neither side is English, and existing bitexts tend to come from narrow domains such as parliament proceedings or subtitles. The authors treat the whole of Wikipedia as a huge comparable corpus. They search it for sentence-level translations across 85 languages, including dialects and low-resource varieties. They consider every language pair rather than only pairs with English.

Sentences are mapped into a single shared space with the language-agnostic LASER encoder. Candidate translations are then selected with a margin-based score, which compares a pair's cosine similarity against each sentence's nearest neighbours. Fast indexing and similarity search make it feasible to process roughly 600 million sentences. The resulting WikiMatrix corpus is released freely.

To check data quality, the authors trained NMT systems only on the mined bitexts and tested them on TED talk test sets, where many pairs reached solid BLEU scores. The resource is especially useful for direct translation between distant non-English languages, since it removes the need to pivot through English.

Key points

  • WikiMatrix contains 135M parallel sentences covering 1620 language pairs from 85 languages. Only about 34M of these sentences are aligned with English.
  • Mining relies on one massively multilingual encoder (LASER, trained on 93 languages) rather than separate bilingual models. Related low-resource languages, such as minority Romance languages like Aragonese or Sicilian, benefit from this shared training.
  • Alignment uses a ratio-based margin criterion with k = 4 neighbours and the "max" strategy. This avoids relying on a single global cosine threshold, which behaves inconsistently across language pairs.
  • For evaluation, fairseq Transformer models were trained for 1886 directions across 45 languages, wherever at least 25k mined sentences existed. Scoring used SacreBLEU on the TED test sets (Qi et al., 2018).
  • 23 systems exceeded 30 BLEU, with the best at 37.3 (pt-br→en). Non-English pairs also performed reasonably, for example Norwegian↔Danish ≈33 and Japanese/Korean ≈17.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

183 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 164 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.05791 · arXiv 1907.05791

@misc{schwenk2019wikimatrix,
  title = {{WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia}},
  author = {Schwenk, Holger and Chaudhary, Vishrav and Sun, Shuo and Gong, Hongyu and Guzman, Francisco},
  year = {2019},
  doi = {10.48550/arxiv.1907.05791},
  eprint = {1907.05791},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.05791}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the Use of Comparable Corpora to Improve SMT performance. In EACL, pages 16–23. find
  2. Sisay Fissaha Adafre and Maarten de Rijke. 2006. Finding similar sentences across multiple languages in Wikipedia. In Proceedings of the Workshop on NEW TEXT Wikis and blogs and other dynamic text sources. find
  3. Ahmad Aghaebrahimian. 2018. Deep neural networks at the service of multilingual parallel sentence extraction. In Coling. find
  4. Mikel Artetxe and Holger Schwenk. 2018a. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. https://arxiv.org/abs/1811.01136. ↗ our card
  5. Mikel Artetxe and Holger Schwenk. 2018b. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. In https: //arxiv.org/abs/1812.10464. find our card
  6. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2017. Weighted Set-Theoretic Alignment of Comparable Sentences. In BUCC, pages 41–45. find
  7. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2018. Extracting Parallel Sentences from Comparable Corpora with STACC Variants. In BUCC. find
  8. Houda Bouamor and Hassan Sajjad. 2018. H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings. In BUCC. http://commoncrawl.org/ ↗
  9. Christian Buck and Philipp Koehn. 2016. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 554–563, Berlin, Germany. Association for Computational Linguistics. find
  10. Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  11. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017. An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification. IEEE Journal of Selected Topics in Signal Processing, pages 1340–1348. find
  12. Thierry Etchegoyhen and Andoni Azpeitia. 2016. Set-Theoretic Alignment for Comparable Corpora. In ACL, pages 2009–2018. find
  13. Simon Gottschalk and Elena Demidova. 2017. Mul-tiwiki: Interlingual text passage alignment in Wikipedia. ACM Transactions on the Web (TWEB), 11(1):6. find
  14. Francis Grégoire and Philippe Langlais. 2017. find
  15. BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora. In BUCC, pages 46– 50. find
  16. Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective Parallel Corpus Mining using Bilingual Sentence Embeddings. arXiv:1807.11906. find
  17. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… find our card
  18. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. arXiv preprint arXiv:1702.08734. find our card
  19. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. https://arxiv.org/abs/1607.01759. ↗
  20. H. Jégou, M. Douze, and C. Schmid. 2011. Product quantization for nearest neighbor search. IEEE Trans. PAMI, 33(1):117–128. find
  21. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit. find
  22. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan M. Pino. 2019. Findings of the wmt 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, Volume 2: Shared Task Papers, Florence, Italy. Association for C… find
  23. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726–739, Belgium, Brussels. Association for Computational Linguist… find
  24. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… find our card
  25. P. Lison and J. Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In LREC. find our card
  26. Mehdi Zadeh Mohammadi and Nasser GhasemAghaee. 2010. Building bilingual parallel corpora based on Wikipedia. In 2010 Second International Conference on Computer Engineering and Applications, pages 264–268. find
  27. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Computational Linguistics, 31(4):477–504. find our card
  28. P Otero, I López, S Cilenis, and Santiago de Compostela. 2011. Measuring comparability of multilingual corpora extracted from Wikipedia. Iberian Cross-Language Natural Language Processings Tasks (ICL), page 8. find
  29. Pablo Gamallo Otero and Isaac González López. 2010. Wikipedia as multilingual source of comparable corpora. In Proceedings of the 3rd Workshop on Building and Using Comparable Corpora, LREC, pages 21–25. find
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  31. Alexandre Patry and Philippe Langlais. 2011. Identifying parallel documents from a large bilingual collection of texts: Application to parallel article extraction in Wikipedia. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 87– 95.… find
  32. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  33. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  34. Philip Resnik. 1999. Mining the Web for Bilingual Text. In ACL. find
  35. Philip Resnik and Noah A. Smith. 2003. The Web as a Parallel Corpus. Computational Linguistics, 29(3):349–380. find our card
  36. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In ACL, pages 228–234. find
  37. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In NAACL, pages 403–411. find
  38. J. Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In LREC. find
  39. Chen-Tse Tsai and Dan Roth. 2016. Cross-lingual wikification using multilingual embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 589–598. find
  40. Dan Tufis, Radu Ion, S , tefan Daniel, Dumitrescu, and Dan S , tefănescu. 2013. Wikipedia as an smt training corpus. In RANLP, pages 702–709. find
  41. Masao Utiyama and Hitoshi Isahara. 2003. Reliable Measures for Aligning Japanese-English News Articles and Sentences. In ACL. find
  42. Yinfei Yang, Gustavo Hernández Ábrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In https://arxiv.org/abs/1902.08564. — [arXiv:1902.08564](https://arxiv.org/abs/1902.08564) ↗
  43. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In LREC. find our card