Elamur Research
☰
home / papers / 2019 / Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1309.pdf

Summary

Neural machine translation depends heavily on the amount and cleanliness of its training bitext. Artetxe and Schwenk therefore address mining and filtering parallel sentences with multilingual sentence embeddings. They train a shared, language-agnostic BiLSTM encoder with an NMT-style decoder. One model covers English/French/Spanish/German on Europarl v7, and a second covers English/French/Russian/Chinese on the UN corpus.

Earlier embedding methods kept nearest neighbours whose raw cosine passed a fixed threshold. The authors show that cosine values sit on different scales for different sentences, so a single cutoff misjudges confidence. Their fix scores each candidate pair relative to the average similarity of its k nearest neighbours in both directions. The best variant uses a ratio and also reduces hubness.

This simple, training-free scoring gives large gains in three settings:

  • the BUCC bitext mining task,
  • reconstruction of the UN corpus alignments,
  • filtering the noisy English-German ParaCrawl corpus for NMT.

The work matters because cheap embedding-based mining with margin scoring can beat heavily engineered pipelines. It was released in the LASER toolkit.

Key points

  • Encoder: a single-layer BiLSTM with max-pooling produces 1024-dimensional sentence embeddings over a joint 40k BPE vocabulary. Margin variants tested were absolute (plain cosine), distance (CSLS-like) and ratio, with k = 4.
  • BUCC: using the ratio margin with max-score retrieval, test F1 exceeds 92 for en-de, en-fr, en-ru and en-zh, peaking around 95.6 for en-de. This is more than 10 F1 above prior published systems.
  • UN corpus reconstruction (11.3M pairs): P@1 reaches 83.27 (en-fr) and 85.78 (en-es), versus 48.90 and 54.94 for Guo et al. (2018). The authors trained on about 2M Europarl sentences, compared with over 400M in-house sentences for Guo et al.
  • ParaCrawl en-de: preprocessing shrank the corpus from 4.59B to 64.4M pairs, and the top 10M pairs by score were kept. This gave 31.2 tokenized BLEU on newstest2014, versus 30.05 for BiCleaner v1.2 and 24.78 for Zipporah v1.2. Adding the remaining WMT data raised the score to 31.8.
  • Release: the code is part of LASER, together with an additional encoder covering 93 languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

147 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 137 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1309

@inproceedings{artetxe2019margin,
  title = {{Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings}},
  author = {Artetxe, Mikel and Schwenk, Holger},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1309},
  url = {https://doi.org/10.18653/v1/p19-1309}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv:1803.05567. find our card
  2. Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the Use of Comparable Corpora to Improve SMT performance. In EACL, pages 16–23. find
  3. Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted Transformer Network for Machine Translation. arXiv:1711.02132. — [arXiv:1711.02132](https://arxiv.org/abs/1711.02132) find
  4. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2017. Weighted Set-Theoretic Alignment of Comparable Sentences. In BUCC, pages 41–45. find
  5. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2018. Extracting Parallel Sentences from Comparable Corpora with STACC Variants. In BUCC. find
  6. Houda Bouamor and Hassan Sajjad. 2018. H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings. In BUCC. find
  7. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. In ICLR. find
  8. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In EMNLP, pages 489–500. find
  9. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017. An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification. IEEE Journal of Selected Topics in Signal Processing, pages 1340–1348. find
  10. Thierry Etchegoyhen and Andoni Azpeitia. 2016. Set-Theoretic Alignment for Comparable Corpora. In ACL, pages 2009–2018. find
  11. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In ICML, pages 1243–1252. find our card
  12. Francis Grégoire and Philippe Langlais. 2017. find
  13. BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora. In BUCC, pages 46– 50. find
  14. Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective Parallel Corpus Mining using Bilingual Sentence Embeddings. In WMT, pages 165–176. find
  15. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Huda Khayrallah and Philipp Koehn. 2018. On the Impact of Various Types of Noise on Neural Machine Translation. In WNMT,… find our card
  16. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit. find
  17. Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In WNMT, pages 28–39. find
  18. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Computational Linguistics, 31(4):477–504. find our card
  19. Dragos Stefan Munteanu and Daniel Marcu. 2006. Extracting Parallel Sub-Sentential Fragments from Non-Parallel Corpora. In ACL, pages 81–88. find
  20. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT, pages 1–9. find our card
  21. Philip Resnik. 1999. Mining the Web for Bilingual Text. In ACL. find
  22. Holger Schwenk. 2018. Filtering and Mining Parallel Data in a Joint Multilingual Space. In ACL, pages 228–234. find
  23. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In NAACL, pages 464–468. find our card
  24. Lei Shi, Cheng Niu, Ming Zhou, and Jianfeng Gao. 2006. A DOM Tree Alignment Model for Mining Parallel Data from the Web. In ACL, pages 489– 496. find
  25. Masao Utiyama and Hitoshi Isahara. 2003. Reliable Measures for Aligning Japanese-English News Articles and Sentences. In ACL. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000–6010. find
  27. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  28. Hainan Xu and Philipp Koehn. 2017. Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora. In EMNLP, pages 2945– 2950. find
  29. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In LREC. find our card
  30. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora. In BUCC, pages 60–67. find
  31. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the Third BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora. In BUCC. find