📄 Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

Summary

Neural machine translation needs large amounts of clean parallel text. Earlier mining pipelines were heavy, distributed and hand-engineered. The authors, from Google AI, propose a lightweight alternative based on bilingual sentence embeddings. Each language has its own Deep Averaging Network (DAN) encoder, which averages learned word and bigram embeddings and passes them through a feedforward network with residual connections. A candidate pair is scored by the dot product of its two vectors, so large-scale nearest-neighbour search becomes practical. Training treats the other pairs in a mini-batch as negatives. The key addition is "hard" negatives: sentences that a baseline model ranks highly but that are not real translations. These force the model to separate true translations from sentences that are only on the same topic. Experiments on en-fr and en-es show that hard negatives consistently beat random ones. The method reconstructs much of the United Nations Parallel Corpus, and when adapted to documents it matches a far costlier system. NMT models trained on the mined data come close to models trained on the gold data. This makes cheap, scalable bitext mining and filtering a practical option.

Key points

• Sentence-level reconstruction of the UN Parallel Corpus reaches P@1 of 48.9% for en-fr and 54.9% for en-es. Hard negatives were computed for only 20% of the training data to save compute.
• At the document level, matching accuracy is comparable to the much heavier approach of Uszkoreit et al. (2010).
• NMT systems trained on the reconstructed pairs score within 1–2 BLEU of systems trained on the original corpus.
• On ParaCrawl, the model's scores correlate only modestly with Zipporah (Pearson's r ≈ 0.4). Adding the model-filtered ParaCrawl data to WMT data gave the best BLEU, about 0.5 above alternatives.
• Translation professionals rated more of the model's selected pairs as GOOD than Zipporah's selections: 76.0% vs 72.0% for en-fr and 74.5% vs 74.0% for en-es. Common errors were partial translations and pairs with identical text on both sides.